1n73r.net

A New Way to Skip Commercials

If you have never owned an HTPC with the ability to automatically skip commercials then you probably should stop reading this post right now. Otherwise, carry on and I will explain to you how I think commercial skipping should work.

How Things Work Now

Currently if you want to flag commercials, you need to record the program and then set up your commercial skipping software to evaluate your recording. The flagging software will look for things like black frames, station logos, volume fluctuation, and other differences that can use to know when there is a commercial and when your show is on. Some software will even allow you to set up profiles on a show by show basis if there is a pattern to when they show commercials. This method works fairly well but there are some major drawbacks that will never be addressed with this model.
The flagging isn’t always 100% accurate
Despite getting better all the time, there are still some tricks that stations do that throw off commercial flagging software. One of the worst in my experience is playing the last joke of the show while the credits are rolling.
Commercial flagging takes a while
Not only does the flagging take a while because every frame of the show needs to be analyzed but the process is also very dependent on how fast your CPU is. If you are flagging commercials while trying to watch TV this can slow down other things drastically. While the process can be sped up a bit with software optimization it will never be instantaneous and this will always be a problem.
It is difficult to flag commercials in real time
If you have a fast enough CPU you can do this with some software but in most cases trying to flag commercials in real time will slow down your system and cause either the CPU to be used 100% or your hard drive won’t be able to keep up and your system will start to freeze or stutter during playback. Just imagine if you were trying to watch live TV, while flagging commercials, while another recording was happening with its own commercial flagging running. Your hard drive would be writing two shows while simultaneously reading 2 shows. If you do want to flag commercials in real time it is going to cost you extra money in hardware.
Commercial info doesn’t sync between all devices
If you have more than one TV your second HTPC should have all the information of the first, but what if you want to take that recording on your laptop, phone, or tablet? Most likely the video player on that device won’t support commercial skipping. The only way to get around this is to transcode the video, but if your flagging isn’t 100% accurate you can’t trust removing parts of the show until after you have watched it. And who wants to watch a show, set the flagging information correctly, transcode the show, and then transfer the show to your mobile device so you can enjoy it?

The Solution

So how can we fix the problem? Crowd sourcing. This would work by having people scan their recordings like they currently do, but it would also give them an option to upload their commercial timestamps to a centralized server.
This server will store the show ID, channel ID, commercial times, and location (time zone). I am not sure if all of that information is needed but it seems like it would cover scenarios of different stations cutting up shows for first time airs as well as repeats and also should cover any differences based on location. The more people that upload their timestamps the more reliable the information will be because the backend can compare uploads to see which ones are consistent and which ones may have missed a part of the show.
With this model all of the above problems could be taken care of because you now can trust your commercial flagging 100% and can even flag commercials in real time on a slow machine. Transcoding would be accurate which would let you transcode with confidence and skip commercials on the go.
Because most HTPC software allows for plugins, there would need to be plugins written for the major HTPC software or current software could be made to adapt to this method. In either case, downloading an XML file to be used for commercial skipping is vastly easier to do than creating the video processors that exist currently. With this method, commercial flagging can be cross platform without the need of video analyzers to detect commercials.

How to Make it Free

This model will be sustained by allowing people to upload their own commercial flags in exchange for being able to download commercial flags. The upload does not need to be done immediately after the show airs and this will allow for commercial flagging to take place during downtime without affecting live TV on the HTPC.
If someone does not have a machine capable of flagging commercials, they can purchase commercial flags for a cheap fee, something to help cover server costs.

I don’t have the time, nor the skill to set up this system so I am throwing it out there for anyone to take advantage of and make the HTPC world a better place. If you do create something that can do this, I only ask that you make a MythTV plugin and let me use the service for free.

Errors with MiroBridge

For a while in MythTV 0.22 I was getting an error with MiroBridge saying Miro was not set up properly even though Miro worked just fine and I triple checked all my config files. I waited until upgrading to MythTV 0.23 to fix it because there was a much easier one button install option in Mythbuntu Control Centre. MiroBridge ran successfully once but I started getting files added to my recordings screen that didn’t exist. Once I manually cleared out the files MiroBridge automatically ran again but caused even more problems by having the two non-recordings show up 44 times.

Once I got all of those recordings removed MiroBridge would never successfully run for me again. I could run MiroBridge manually but every time I would get an error

AttributeError: ‘NoneType’ object has no attribute ‘lower’

and MiroBridge would just hang. I would have to force quit the script and never got past that point. I figured out how to fix this issue with a lot of Googling. The problem happens because all of those files that didn’t exist from before were referenced in my database and MiroBridge was trying to remove them. Because they weren’t cleanly removed from the recordings screen (or Miro) MiroBridge couldn’t delete the listings. So what we have to do is clean up these erroneous database references manually. If you have phpMyAdmin installed on your machine you can easily delete these referenced recordings through there but I didn’t want to have something else installed on my MythTV box so I did it through the terminal. Before you go on this is a great chance to learn how to use screen because we are going to want to reference a lot of information from a bunch of different terminal windows. You can either do this with multiple terminals or you can be cool and do it with screen.

Here are the steps you need to take to fix this error.

Step 1. Troubleshoot and Backup
Close out of MythTV and open a terminal. From the terminal run MiroBridge with the simulate option (-s) and redirect the output to a file (>). In MythBuntu 10.04 the command is
/usr/share/doc/mythtv-backend/contrib/imports/mirobridge/mirobridge.py -s > mirobridge.log
BTW this command needs to be run from the local machine and not through ssh because I believe the $DISPLAY variable isn’t set to :0 so MiroBridge gets confused when you are remote. We could change that variable but it is just as easy to work from the local machine.
Now we need to make sure you are having the same error I was having. To do this we are going to read that file you just created and check for any lines that tell MiroBridge to delete old recordings from the database.
cat mirobridge.log | grep oldrecorded
If you don’t have any output from this then you probably have a different error than what I was getting. If you have a bunch of lines outputted that look like this

Simulation: Remove orphaned oldrecorded record (None – ?????)

Then you have the exact same error and should continue on to the next step. Make note of the ????? value cause we will need it later. It should be the name of the video file that was giving us errors.
It is a good idea to always backup your mythconverg database so you may want to run this before continuing
/usr/share/mythtv/mythconverg_backup.pl

Step 2. Close Miro
ps aux | grep miro
Find each line that says something like /usr/bin/miro or mirobridge.py and then get the number at the front of that line and type kill *number* If you have more than 1 line just put each number and it will kill all of the running instances.
Now that Miro and MiroBridge have been killed move on to manually removing the recordings information form the database.

Step 3. Remove old data
First thing you will need is your database username and password. The easiest way to get this is just to use
cat /etc/mythtv/mysql.txt
At the begining of the output you should have DBUserName and DBName. You will need both of those but it is probably mythtv and mythconverg so make sure you also get the DBPassword near the bottom of that output.
Once you have that information connect to your mysql server. Replace ‘password’ with your actual password from the /etc/mythtv/mysql.txt file.
mysql --user=mythtv --password='password'
Then we just want to make sure we have the right mysql server (just because it is better to be safe than sorry).

SHOW DATABASES;
You should see your mythconverg table here so go ahead and use it.

USE mythconverg;
SHOW TABLES;
You should then see a whole lot of tables but the one we want is the oldrecorded table. Just to make sure we have the right information you are going to view all the erroneous database entries before you delete them. For the ????? you need to use the recording name that you got from step 1. Make sure you use the correct case when searching for the information.
SELECT * FROM oldrecorded WHERE subtitle = "?????";
If there were more than one different entry names simply put OR subtitle = “?????” with your next value. In my case I had two different recordings that caused the problem so I used
SELECT * FROM oldrecorded WHERE subtitle = "STALKER_REPORTER_ITUNES.mp4" OR "tekzilla--tzdaily--0464--2009-10-16gmaps--hd.h264.mp4";
This should list all of the entries so now you just need to delete them from the database.

DELETE FROM oldrecorded WHERE subtitle = "?????";
Just like above you can use the OR command to delete multiple entries at once.
Once the entries are deleted just type exit; to leave mysql and go back to the normal shell prompt.

Step 4. Test
Now that the the entries have been deleted repeat step 1 again and you shouldn’t have any lines returned that say
Simulation: Remove orphaned oldrecorded record (None - ?????)
You should then be able to run MiroBridge without the simulation option and get your Miro videos imported into your recordings. It is a good idea to still use the verbose (-V) option just to make sure everything goes smoothly.
/usr/share/doc/mythtv-backend/contrib/imports/mirobridge/mirobridge.py -V

That is all you need to do. Just make sure you set up your cron job to automatically run MiroBridge however often you want and you should be all set.

I hop this helps and leave a comment below if you have any questions.

MythTV Star Trek Theme

The MythTV theming challenge comes to a close Sunday January 31st and the entries are starting to show up on the MythTV theming mailing list.
Here is one that showed up that I am very excited about. It was created by Gary Butters (not me).

If you already have MythTV set up you can grab the theme from here. Just extract the folder from the tar.gz to the /usr/share/mythtv/themes/ folder and it should show up in MythTV frontend under setup -> appearance.

MythTV how-to video – Part 2

This is a continuation of Part 1 and in this video I just walk you through a Mythbuntu installation and what options you want to use. The next parts will be about configuring the software past the initial installation.

Sorry for the sound quality. I had to use my digital camera for video (which is what I used in part 1) but for some reason there was a ton of static on the recording.

MythTV how-to video – Part 1

I have been working on my MythTV box for some time now and finally got around to editing the first part of the how to I recorded when setting it up. The first part is just about setting up the hardware (not much new), but the second part (and probably third and forth) is going to be on setting up the software and getting things configured just the way you want them to be.

Mythtv Log

I decided I needed a better way to keep track of myth MythTV systems I have used in the past, and what I needed to do to get the systems up and running. I started a MythTV Log page that should show up on the left (or the links section depending on my skin). I mainly posted this to help other people that are setting up their own MythTV computers using similar hardware. Check out the hardware I am using, and have used in the past, by looking through the logs.
Let me know if you have any good tips on keeping installation logs organized. I am fairly new to all this.

Compile Lirc on an AOpen mp945-dr with Ubuntu

I recently traded in my Mac Mini for a AOpen mp945-dr to see if it would work any better with MythTV and Boxee. I started off with Ubuntu 8.10 installed but found some random problems with video not displaying every time the computer would start so I went back to 8.04. On problem I had with both installations was that the remote sensor did not work out of the box. I did some digging and found that Lirc does work with this machine but the latest development branch was not included in Ubuntu for stability reasons. Here are the steps I had to take to get the remote sensor working.

As a overview we are going to:
1. Download the tools needed to compile software in Ubuntu.
2. Download the latest version of Lirc using CVS
3. Compile Lirc source
4. Test it real fast to make sure it works.

If you have Jaunty Jackalope (9.04) see the notes at the bottom.

1. Download the needed compiling tools.
sudo su
apt-get install libtool autoconf automake linux-headers-`uname -r`-generic cvs
This will download and install the compiling tools.

2. Download Lirc
First lets make a folder to put it in

mkdir ~/lirc
cd ~/lirc

Then we will download the newest source from sourceforge. We are using a tool called CVS to read more about it check out here.
cvs -d:pserver:[email protected]:/cvsroot/lirc login
cvs -z8 -d:pserver:[email protected]:/cvsroot/lirc co lirc
Because we ran that command while in the ~/lirc folder everything downloaded there so we just need to run the commands to configure everything before we build it.

./autogen.sh
./setup.sh

In the graphical interface I selected Driver config > USB > mceusb2 (new)

Then continue with compiling

make
make install
modprobe lirc-mceusb2

Because Ubuntu does things slightly different we need to copy a couple files to where Lirc expects them to be.

cp /lib/modules/`uname -r`-generic/misc/lirc_dev.ko /lib/modules/`uname -r`-generic/ubuntu/media/lirc/lirc_dev
cp /lib/modules/`uname -r`-generic/misc/lirc_mceusb2.ko /lib/modules/`uname -r`-generic/ubuntu/media/lirc/lirc_mceusb2

Then we can continue with installing the new Lirc into the running kernel.
rmmod lirc_mceusb2
rmmod lirc_dev
lsmod|grep lirc
/etc/init.d/lirc restart
Everything should be compiled and in place now.

4. To test it out run
irw
and then push some buttons on the remote. You should see the commands you press displayed in the terminal. If you see that then you can just push Ctrl+C to stop irw and you should be all set to go.

I had to change this tutorial slightly from where I originally posted it in the Ubuntu forums so that it would be more universal. If you have problems please leave a comment and check out the original thread here.

I hope this helps.

For Jaunty (9.04) I did not need to install the headers (it actually failed because it was already the newest version) I also needed to install “dialog” and “build-essential”. I had to create the “/lib/modules/`uname -r`/ubuntu/media/lirc/” folder before I could copy the configurations, and the folder I created also changed slightly to cp /lib/modules/`uname -r`-generic/misc/lirc_dev.ko /lib/modules/`uname -r`-generic/ubuntu/media/lirc/lirc_dev
cp /lib/modules/`uname -r`/misc/lirc_mceusb2.ko /lib/modules/`uname -r`/ubuntu/media/lirc/lirc_mceusb2
More updates as I find them.

Well that was stupid…easy MythTV fixes

Just a real quick tip for anyone having problems watching live TV or recording shows in MythTV. After a fairly fresh install I could not watch live TV. My screen would go black for a second and I would be kicked back to the main menu. I also was not able to record shows. The recording would show up on the schedule but nothing would happen when the time to record would actually come. So my tip is check your log files!!! They are stored in /var/log/mythtv (at least they are in ubuntu). You should have mythbackend.log*, mythwelcom.log, and mythfrontend.log* in that directory. The log files are incremented making the highest number the oldest file and the one without a number your newest file. So check the newest file first.
As for my problem I checked my frontend log first, but I didn’t see much information besides the fact that I changed skins. So I decided to check the backend. Here is what I found.

2008-11-23 10:26:22.432 TFW, Error: Opening file '/media/mythtv/recordings/1941_20081123102621.mpg'.
eno: Permission denied (13)
2008-11-23 10:26:22.436 TVRec(1) Error: RingBuffer '/media/mythtv/recordings/1941_20081123102621.mpg' not open...
2008-11-23 10:26:22.437 TVRec(1) Error: CreateLiveTVRingBuffer() failed
2008-11-23 10:26:22.438 TVRec(1) Error: Failed to create RingBuffer 1

Now this may not be too obvious to most people but take a look at “Permission denied (13)”. DOH! I forgot to give my user permissions to the directory I set up to record my shows in (as well as my live TV folder).

sudo chmod 777 /media/mythtv/*

and now I am able to watch live TV and record whatever shows I feel like.
Just thought I would save you the hassle if this happens to you.

Why the Mac Mini sucks for MythTV/PVR

I bought the mac mini for a few reasons:
1. It is very small and very quiet
2. It looks good with my AV equipment
3. There are a lot of accessories that match the looks of the mini
4. It came with good enough specs for video playback and recording

The main problem I had was dealing with the small/slow hard drive. So I wanted to upgrade that but here are some of the things I ran into.

First of all, the Mac Mini had almost everything I wanted (and some things I didn’t care about) built in. I wouldn’t need to add any features to the device unlike other small computers I was looking at. Most other small computers came with almost everything I needed except one or two thing, built in IR for a remote or firewire. These were requirements for me. I know I could have used a USB -> IR adapter but I wanted this computer to look a little less DIY than my last MythTV. And not having firewire seemed too complicated to try and add my own through expensive adapters.

My goal for the Mac Mini was to upgrade the internal hard drive and use an external 1 TB drive. The reason for this was storage and speed. The internal hard drive in the Mac Mini is only 5400 rpm and while that works fine for web pages and word documents, when reading and writing large media files all day it just would not be fast enough for the task. One option was to install the OS on the internal hard drive and use a external hard drive just for storage over USB but the CPU overhead of USB and the read/write speeds would just not cut it (internal SATA II is over 6 times faster than USB 2.0/Firewire 400). There also is no firewire 800 port so that wouldn’t work either. I tried following a walk through that someone else already did with a mini hard drive upgrade but apparently Apple has updated their device since this was done and adding a hard drive the same way is no longer possible. If this worked for you on a post Nov 2007 Mac Mini please let me know in the comments. The problem I had was the Mac Mini would never detect any hard drive outside of the chassis. I tried with multiple cables, hard drives (2.5″ and 3.5″) but never once was able to get OSX or Ubuntu installed on the bigger hard drive.

To break it down, the reasons the Mac Mini actually sucks for building your own PVR are these.
1. Slow (low storage) hard drive
2. Expensive features that are not needed but you don’t have a option but pay for. (OSX, iLife, Apple tax, etc.)
3. Hard to upgrade processor and memory

And these are the reasons I picked the AOpen over the Mac Mini.
Mac Mini uses a older 945GM chipset with 667 Mhz vs. GM45 chipset with 1066 Mhz, it uses a more power hungry 65nm (T5600/T7200) processor vs. 45nm (T8100 or any socket p processor you want), and GMA950 GPU vs. GMA X3100. Not really big deals considering both should have the ability to play back most 1080p content. But having updated specs just make things a little easier, and the ability to upgrade helps make the PVR future proof…well almost.

For anyone interested, here are some pictures of the steps I took to make the Mac Mini NOT work with an external eSATA 1 TB hard drive.

	Sata to eSATA adapter. I was sent the wrong adapter and Microbarn would not accept the fact that this was a female SATA to male eSATA cable when I needed male to male. Even after sending them this picture. P.S. don’t order this, order this.
	Adapter plugged into the Mac Mini SATA riser card with a male to male sata adapter I had to buy extra thanks to Microbarn. Maybe I just don’t understand male and female connectors.
	SATA cable run inside the case. This wasn’t as hard as I thought it would be. See the picture above for what it looked like coming out of a slightly modified Kensington lock hole.

Let me know how a Mac Mini is working out for you as a PVR or what other machine you bought as a substitute in the comments.

MythTV…another build

So I have really been missing the ability to record/rewind/pause TV. I haven’t had a full time MythTV box for over a year and a half. I sold my old MythTV computer and had been saving the money till something perfect came out to use as a fast and quiet MythTV computer and samba server. I actually bought a Mac mini about 2 months ago in an effort to get it working with a external hard drive for speed and storage. Needless to say, Apple did a good job of making the Mac mini too annoying to try and get it to work the way I wanted it to. I will post a little on my experience there later.
So I returned the mini and found the AOpen mp45-dr. It is about the same size as the mac mini (technically smaller) and it came bare bones. This is great because I could either get the same specs as the mini for cheaper or I could improve the parts I wanted to improve and not get the “features” I didn’t need (like internal wifi or internal laptop hard drive). Unfortunately, the mp45 had one thing I really needed to complete my HTPC, firewire. I looked around to see if I could get firewire out of the eSATA port or one of the two internal PCIe mini ports but I couldn’t find anything that would work the way I wanted and I needed firewire to record premium HD shows from my cable STB. So I found the mp945-dr which is just the older model of the same computer. It looks almost exactly the same but has a slightly slower video card, no eSATA port, and slower chipset (which included a slower FSB). When I really looked at it though, the parts it came with would be plenty for what I needed and so I used the money from my MythTV sale (and a few other side projects) and bought the AOpen mp965-dr.
Here is my build.
AOpen mp965-dr (from Buy.com)
Intel Core 2 Duo T8100 (from newegg)
2GB Kingston Ram (from newegg)
WinTV-HVR-950 Tuner (from newegg)
1TB ministack v3 (from other world computing)
Sata -> eSATA cable (from mwave)
I also already had a HDHomerun tuner and firewire cable for a tuner from my STB.

I am excited to get this up and running again and will post my build as time goes on. Here is a quick rundown/reference to the sections I will be writing. I will update these later with links.
Part 1: Setting up the hardware
Part 2: Choosing the software
Part 3: Setting up software
Part 4: Using the machine day to day