Playitsam: Play, Edit & Export TiVo TyStreams

Version 1.83, September 2002. (c) Warren Toomey, GPL license.

This program allows you to edit a recording made on a 1.3 or 2.5.1 Standalone TiVo, and to export the edited stream to an external storage. From there you can either play back the recording on the TiVo, or split the recording into MPEG audio and video streams.

1 Features

Streams can be edited either on the TiVo itself, or on a separate Linux/Unix machine.

Only the edited portion of the stream needs to be exported from the TiVo; you don't need to extract the whole recording to edit it.

Streams can be played back from the TiVo MFS (native) area, from a Unix file (both TiVo and Linux/Unix), or from a remote `tystream' server over TCP.

Streams can be exported to a Unix file, or to a remote `tystream' server.

The tystream server can be a Unix/Linux machine or a TiVo. This means that you can play a recording made by one TiVo back on another, as long as they both have Ethernet connections.

Edits are done on Group of Picture (GOP) boundaries. This means that the edited tystream can then be manipulated with TcConvert, tyc or vsplit to produce valid MPEG-2 files.

There is on-screen (i.e on TV) user feedback of the editing process.

2 Differences from Earlier Versions

The main differences from Playitsam v1.74 and this version are:

Playitsam now (nearly) gets the audio editing correct.

There is a new ttyserver which runs on the TiVo and acts like tyserver, except that it can send arbitrary chunks.

Vsplit now also helps to keep the audio in sync. You will find that the extracted audio is IN SYNC, and there is no need for video or audio offset delays.

MyWorld is not restarted, as it only worked in version 1.3. The -M and -n options are gone; so too is input from stdin.

There are a few extra tools and scripts.

3 Acknowledgements

Playitsam is based on the vplay program written by Andrew Tridgell; in fact, I have converted his MFS routines into a library. Because of this, Playitsam is released under the GPL license. The on-screen display routines come from tivovbi by Embeem. David Gerdes provided lots of useful feedback, and the code to link libmpeg2dec into Playitsam.

The program was inspired by an executable-only program called Playback which was released by `the Doctor' in the Dealdatabase TiVo Hacks Forum. I wanted an open-source version, so I wrote Playitsam.

4 CHOOSING THE EDITING PLATFORM

You need to decide if you want to edit your TiVo recordings on the TiVo itself, or on a separate Unix/Linux machine. The options are:

There are advantages and disadvantages to both. If you choose to edit using the TiVo:

You will need to stop the normal TiVo user interface while editing, and reboot the TiVo once the editing is finished.

You can use the TiVo remote control to send the editing commands to the Playitsam program.

You will need another machine running the `tystream' server so that you can save the recordings. The tystream server can run on a Unix/Linux box or on a Windows box.

If you choose to edit on a Unix/Linux box:

You will need to compile Playitsam for this machine.

You will need to have mpeg2dec and mpg123 installed, so you can see the video and hear the audio.

You will send keyboard commands to Playitsam to do the editing; there is no glossy GUI.

You will run the `tystream' server on the TiVo itself, and you won't need to stop the TiVo user interface nor restart the TiVo.

5 INSTALLATION ON THE TIVO

If you have chosen to use the TiVo itself to edit your recordings, then this section describes how to install the program in the TiVo.

Playitsam comes as a gzipped tarball which contains the source code, this documentation and a TiVo binary called tplayitsam. I've only called it this so that if you accidentally type make to make the non-TiVo version of the program, it won't overwrite the TiVo version.

You should copy tplayitsam over to your TiVo to wherever you have installed your other binary tools. Remember to chmod it 755 so that it is executable.

6 INSTALLATION ON UNIX

You will need to compile playitsam if you are going to edit on Unix/Linux. You may also want to compile the tyserver program for your Unix box, which allows you to make your box into a TiVo stream server.

Simply type make after unpacking the files from the Playitsam tarball. This will produce a binary called playitsam which you can copy to /usr/local/bin on your system. The make will also make the tyserver program. Again, copy it into /usr/local/bin. Make sure both are executable with a chmod 755 on both.

The make will also try to compile tplayitsam, and will fail if you don't have a TiVo cross-compiler installed. This isn't a problem because I have provided the tplayitsam binary. See the Makefile if you do want to recompile the TiVo binaries.

At present, I am using mmap() in the tyserver program to speed up I/O across the network. If your Unix system doesn't support mmap(), then remove the `#define USING_MMAP' line in tyserver.c to turn off the use of mmap().

To run playitsam on Unix/Linux, you will need the mp2 audio playback tool mpg123, and the MPEG video playback tool Mpeg2dec installed.

6.1 Playitsam and libmpeg2dec

The make of Playitsam above uses the standalone mpeg2dec application. However, David Gerdes sent me some code diffs to allow Playitsam to use the mpeg2dec library. This is considered experimental! Read the beginning of the Makefile to see how this is done.

It appears that the mpeg2dec library consists of at least two files, libmpeg2dec.a and libvo.a. I also get libcpuaccel.a on my system. On your system, perhaps libmpeg2dec.a and libcpuaccel.a are merged? You might also file shared libraries (.so files) instead of archive libraries (.a files).

The set of X Windows libraries required also seems to be different on different platforms. Some systems need to have -lSDL -lSM -lpthread, but on FreeBSD these don't seem to be required.

In other words, you should have some knowledge of Unix libraries and Makefile rules if you want to go down this route of operation! I would suggest not trying this at all.

7 INSTALLATION OF THE TYSTREAM SERVER

There are basically two modes of Playitsam operation:

You edit on the TiVo itself, and export to a tystream server either on Unix/Linux or on Windows.

You run a tystream server on the TiVo itself, and import the stream from the TiVo while you edit it on a Unix/Linux box.

With option 2, when you unpack the Playitsam tarball, you will find a file called ttyserver. This is a TiVo binary tystream server. Copy it over to your TiVo, chmod it to 755 and run it by doing:

bash# ttyserver

If you choose option 1 and want to run the tystream server on a Windows machine, then you can obtain the tyserver.exe binary from http://minnie.tuhs.org/Programs/Playitsam/tyserver.exe. Many thanks to Grant Stockley for compiling the program under Windows.

If you choose option 1 and want to run the tystream server on Unix/Linux, then simply type make after unpacking the files. This will produce a binary called playitsam and will also make the tyserver program. Copy both into /usr/local/bin. Make sure both are executable with a chmod 755 on both. Then you can run tyserver in the background by doing:

% tyserver /directory/name &

Unlike the TiVo version, the Windows/Unix/Linux tyserver requires a directory name. The directory should exist, and obviously should be writable by you if you want to export streams from the TiVo to this directory. The tyserver will open up TCP port 1150, and wait for commands from the TiVo. You need to know the IP address of the machine running the tyserver, so that you can reference it when starting Playitsam on the TiVo.

There are only 3 commands understood by the tyserver at the moment. LIST will get a list of files from this directory, and Playitsam doesn't use this at present. SAVE is sent by Playitsam when it goes into Record mode, and names the new stream's filename. Straight after that, the stream is sent down the TCP connection. Finally, the PLAY command is sent when Playitsam goes into Play mode; an existing file is named, and the tyserver sends this file back to Playitsam for playback. You can of course nominate a read-only directory like /cdrom, but you will only be able to play streams back from this directory.

For security reasons, only run the tyserver or the ttyserver on a private network which is not accessible from elsewhere. There is no authentication with the tyserver or the ttyserver, and a malicious person could copy your TiVo recordings or fill up your hard disk with streams!

8 RUNNING THE PLAYITSAM PROGRAM

Playitsam is run on both TiVo and non-TiVo systems as follows:

% playitsam [-v] [-p] [-D] [-c] inputsource [outputsource]

The run-time options are:

-v   Be verbose when running. It's only really useful for debugging Playitsam.
-p  Only play the stream back, don't go into editing mode.
-D  On the TiVo, disable on-screen feedback of the editing process.
-c  On Linux/Unix where you have compiled with the mpeg2dec library, this disables the use of any Intel CPU acceleration techniques when the video is being played. Experimental.

When editing on the TiVo, Playitsam will kill off the EventSwitcher (i.e the TiVo user interface) at startup. If you are using Playitsam on the TiVo, then before you run it, you should make sure that there is no writing on the screen. Either go to Live TV and clear the green bar, or start playing an existing recording. If you don't, then you will have to put up with writing on the screen! You will need to reboot your TiVo once the editing has been completed.

Playitsam normally goes directly into Edit mode. However, if you choose -p, then Playitsam will go straight into Play mode and play the stream; no editing will be allowed.

8.1 Input and Output naming Conventions

The input source follows this naming convention:

If the input source is a number, then it is the FSID (stream identifier) for a recording on the TiVo. You will need another tool to get a list of FSIDs for your recordings. I have included a Tcl script called nowrunning.tcl which you can run on your TiVo to get a list of FSIDs.

If the input source is a set of numbers separated by commas, then this is a list of FSIDs which make up the overall recording, e.g 5125,5126,5127. There are no spaces in the list, just digits and commas.

If the input source has a colon in it, then the stream will come from a tystream server over the network. The bit before the colon is the tyserver's IP address. The bit after is the stream name on the tyserver, e.g 10.10.1.1:fred.ty or 10.10.1.1:5125,5126,5127.

Finally, if none of the are true, then the stream will come from a Unix file on the TiVo. This can include an NFS or SMB mountpoint.

Here are some examples:

# tplayitsam 45887 # TiVo FSID for stream

# tplayitsam 1123,1124,1125 # Multiple FSIDs

# tplayitsam 10.10.1.3:the_bill.ty # Stream from a tyserver

% playitsam 10.10.1.1:1234,1235 # Stream from a TiVo tyserver

% playitsam ../my_hero.ty # Unix file

The output source is optional, and will only be used if you go into Record mode (see below). The name for the output file follows the same naming conventions as above:

An IP address, colon, file will save the edited stream to the tyserver as the named file. The IP address you use is the IP address where the tyserver program is running, not the TiVo's IP address.

Otherwise, the output will be saved as a Unix file.

There is no option to save the stream back into the MFS.

So, here are some example Playitsam command lines. Remember that tplayitsam is where we are editing on the TiVo itself, playitsam is where we are editing on Unix/Linux:

# tplayitsam 1123,1124 fred.ty # Edit 1123,1124, save as fred.ty

# tplayitsam 1123,1124 10.10.1.3:fred.ty # Save to external tyserver

# tplayitsam -p 10.10.1.3:fred.ty # Playback recording from tyserver

% playitsam 10.10.1.1:1123,1124 fred.ty # Edit stream on remote TiVo, save as fred.ty on Linux

9 EDIT MODE

In Edit mode, you can go forwards and backwards through the stream, and mark cuts to make. Then, when you go into Play or Record mode, the sections of the recording that you have cut will be skipped.

The original input source is not altered by your editing operations; only the playback of the stream, or the exporting (recording) of the stream will be affected.

Edit mode starts by playing the input stream. A cut is marked starting at the first chunk, but with no end yet. This assumes that your program starts with something you don't want. If not, it's easy to clear this pending cut.

The beginning and ending of a cut can only occur in a chunk in the tystream which contains a Group of Pictures (GOP) marker. Only every second or third chunk has a GOP marker. This is why you won't be able to pause at every frame, nor make cuts on every frame.

You can edit the stream using the Unix keyboard, or the TiVo remote. Here are the commands available on the keyboard:

Keystroke	Operation

space	Toggle pause & playback
f	Go forwards 1 chunk
1,2,3	Go forwards 5, 25, 125 chunks
4,5,6	Go forwards 500, 2500, 12500 chunks
0	Go to the last chunk and pause
b	Go backwards 2 chunks
!,@,#	Go backwards 5, 25, 125 chunks
$,%,^	Go backwards 500, 2500, 12500 chunks
)	Go to the first chunk
c, s	Make a cut starting at this chunk
e	End the current cut at this chunk
k	Kill (delete) the current cut
K	Kill all the cuts made so far
l	List the cuts made so far, and size of edited recording
P	Go into Play mode to play the edited stream
R	Go into Record mode, and export the edited stream
q,Q,ctrl-C	Quit Playitsam

If you are using Playitsam on the TiVo, then you can also use the TiVo remote control to do most of the above editing operations. Here is the mapping of TiVo remote buttons to operations:

When you pause on the TiVo, and assuming that you haven't selected -D, then you will see the chunk number that you are up to. If you start or end a cut in pause mode, then this will be announced on the screen for you.

Information about the chunk numbers and the cuts you make are also shown on the login/telnet connection.

If you start a cut, but do not finish it, then Playitsam will assume that you don't want to record anything from the cut start onwards. This means you don't have to go right to the end of a program just to trim some of the end off.

10 RECORD MODE

Once you have finished editing your recording, you will want to save the edited stream. To do this, you need to go into Record mode. This is done by entering R from the keyboard, or the Record button on the TiVo remote. In Record mode, the stream minus the cut material is exported to the named output source.

Playitsam on the TiVo will print an on-screen message to let you know that it has gone into Record mode. On Unix/Linux, the same will occur, and you will be given details of the size of the recording in megabytes, and the mpeg2dec video window will be closed down.

At every 100th chunk in the original stream, some information is displayed to let you know that the recording is actually working. When the edited stream is finally exported, Playitsam will exit normally.

11 PLAY MODE

If you choose to go into Play mode from Edit mode (with P from the keyboard, or Select on the TiVo remote, or using the -p option), then Playitsam will go to the first chunk and start playing the stream. If playing on the TiVo, it will pause for 10 seconds before starting the playback. During the 10 second initial delay, Playitsam will print out a countdown, so you can work out when the playback will start.

The TiVo remote cannot be used when playing back a stream. However, you can terminate the playback by sending TiVo Central, Select, TiVo Central with the remote.

12 OTHER TOOLS

There are two other C programs which come with Playitsam. The first is called mfs_export, and this is a TiVo program which will export a whole (i.e unedited) tystream to a remote tyserver. The tarball of Playitsam includes a TiVo binary for mfs_export. You can run it as:

# mfs_export 1234,1235,1236 10.10.1.1:fred.ty

The other is a tool called typarse, which reads a tystream and prints some diagnostic information about the recording. There is also an associated shell script called timeparse, which prints out the timestamps for the audio and video portions in the tystream. Please read the source code to work out how to use them.

13 CONVERTING EDITED STREAMS TO MPEG FORMAT

The TiVo native video stream format (aka tystream) is a proprietary multiplexed MPEG-2 stream. This has to be split into video and audio streams (.mpv and .mpa files), and then re-multiplexed into true MPEG-2 format.

The Playitsam tarball includes the source to a program called vsplit, which will split the tystream into separate files. You can compile it for your Linux/Unix box, or you can obtain the Windows binary from http://minnie.tuhs.org/Programs/Playitsam/vsplit.exe. To run the program on an exported tystream, simply do:

% vsplit input.ty output.mpa output.mpv

where input.ty is the input file, and the other two filenames are the output files. For the Unix/Linux users, I also include a shell script called ty2mpg, which splits an input file using vsplit, optionally re-encodes the audio at 44.1kHz, and them multiplexes the output into the MPEG file. You will need to install these tools for the script to work:

mpg123

Toolame

mplex from the Siemens DVB toolset

The ty2mpg script runs as follows:

% ty2mpg [-n] input.ty

and produces an output file using the input filename but with .ty replaced with .mpg. The -n option tells ty2mpg not to resample the audio stream.

14 REMOVING TYSERVER STUTTERING

You might find that playback from the tyserver or the ttyserver has a lot of stuttering. I have found that there are at least two reasons for this:

non-optimal setup of the TCP options on the box running the tyserver

use of a CD-ROM drive which speeds up & slows down

Firstly, you may need to tune the TCP options on your Unix box so as to increase the TCP receive window and the TCP send window; this helps to keep the TCP stack transmitting and receiving data in the face of the inevitable LAN collisions. On my FreeBSD box and using a 1.3 TiVo, I used these sysctl commands at boot-time:

/sbin/sysctl -w net.inet.tcp.recvspace=65536
/sbin/sysctl -w net.inet.tcp.delayed_ack=0
/sbin/sysctl -w net.inet.tcp.sendspace=131072

These set the TCP receive window to 65,536 and the TCP send window to 131,072, and disable delayed TCP acknowledgements. However, I found that when I upgraded to a 2.5.1 TiVo system, I had to change the sendspace value to 2400. In fact, the sendspace value seems to make a big difference! Going up to 3072, I could only get 0.1Mbps, but with 2400 I am getting 7Mbps over TiVoNet. If you are seeing very poor playback with Playitsam from the tyserver, then I would spend some time tuning your TCP configuration. I don't have a Linux or Windows box, so I can't tell you how to do the same tweaking on these systems. if you can supply details, I would be grateful.

Secondly, you need to get a decent CD-ROM drive, if you are going to play streams back from CD-ROM. I was initially using an 20x ATAPI CD-ROM drive and playback was awful. I found that the drive was continually speeding up & slowing down, and this was causing the stuttering. I now have a 16x SCSI-2 CD-ROM drive, and now I rarely get any stutters on playback.

However, to really get rid of the stutters, use the tyserver and not NFS or SMB, tweak your TCP options, and copy the stream to hard disk instead of playing from CD-ROM.

15 BUGS AND OMISSIONS

This is the end of the line for Playitsam version 1.x. Because of the way tystream chunks are used, there will be some streams which when exported do not produce valid MPEG output. There is no point in trying to fix version 1, but hopefully version 2 will rectify this shortcoming. There is no set date for version 2; in fact I have only sketched out the basics of the next version in my head.

Because of the way the stream is edited on the TiVo, you will find what looks like blocky or corrupted images when you go forward or backward frames. This is normal, and is due to the fact that MPEG-2 streams encode differential images as well as full images. So when you skip data, or play stuff back in the wrong order, then you will get artifacts like this. Hint: if you are in pause mode and are going backwards, go backwards more chunks than you need to, and then come forwards. This helps to bring the video back into `sync' so that it is uncorrupted again.

The Mpeg2dec program on Unix/Linux sometimes dumps core when you jump chunks or go backwards; this seems to occur on occasions when the video stream is slightly corrupted.

If you have any questions or comments, please e-mail me at wkt@tuhs.org.

Warren Toomey, September 2002.

File translated from T_EX by T_TH, version 2.78.
On 5 Sep 2002, 08:07.