Playitsam: Edit & Export TiVo TyStreams

Version 2.00, January 2003. (c) Warren Toomey, GPL license.

This program allows you to edit a recording made on a 2.5.x or 3.0 Standalone TiVo, and to export the edited stream as separate MPEG audio and video files.

1  Caveats and Limitations

  1. You need a separate Linux/FreeBSD box with X11 installed to use Playitsam. It does not currently run on Windoze.
  2. Playitsam version 2 is an editor only; it cannot play back TiVo streams in any decent fashion. It will only export edited TiVo streams as separate MPEG audio and video files.
  3. Version 2.00 is a proof of concept version. The user interface is appalling, and there are sure to be bugs. This is not a production version, and there is little chance it will ever become a production version.
  4. You will also need to install mpeg2dec and mpg123, so you can see the video and hear the audio when you are editing.

2  Unpacking and Compiling

Unpack the Playitsam tarball, and you will find these directories:

If you type `make' from the top-level, it should build libplayit.a, the miscellaneous tools and the command-line version of Playitsam, and leave these in the bin directory.

If you want to build the GTK version of Playitsam, then you should go into the gplayitsam directory and do: ./configure; make. The resulting gplayitsam binary will be left in the src/ directory.

You do not need to build the TiVo ttyserver binary, as there should be a copy in the bin directory. However, if you do want to rebuild this, you should first edit the Makefiles in libmfs and tivo. Then from the top-level, you can make ttyserver.

3  Installing the Binaries

Once you have built the FreeBSD/Linux binaries (getchunks, ntyparse, nvcut, nvsplit, playitsam, gplayitsam), install some or all of them into a suitable place like /usr/local/bin.

You will also need to copy the binary of ttyserver to your TiVo and also install it somewhere on your path, so that you can run it. You might also like to copy the Tcl script nowshowing.tcl to your TiVo; this can be used to determine the FSID list for each program that you have recorded.

For all of the FreeBSD/Linux binaries to work, the ttyserver must be running on the TiVo. Copy it over to your TiVo, chmod it to 755 and run it in the background by doing:

bash# ttyserver

Socket has port #1150

The message `Socket has port #1150' should appear if this is running correctly. Note: the ttyserver will allow any connections to TCP port 1150 on your TiVo. You don't want your TiVo to be directly visible on the Internet when you have the ttyserver running.

4  Cutpoints and Splitpoints

Playitsam uses the concepts of cutpoints and splitpoints. A cut is a section of the program that you want removed. There are two cutpoints: the start of the cut, and the end of the cut. The material between the cut start and the cut end will be removed. Playitsam always starts running with an implicit cut start at position 0, i.e from the beginning of the program.

A splitpoint allows you to save your output into separate files. If you set a splitpoint, then at that point Playitsam will stop writing to your output files, and open up a new set of output files. This allows you to break a very large program into several pairs of MPEG files. Each MPEG file pair will be numbered 1, 2, 3 etc.

You can set a splitpoint at any point in the program, and you can end a cut and set a splitpoint at the cut end. This allows you, for example, to edit out some commercials and also start a new file at that point.

5  Running Playitsam

Playitsam is the command-line version of the editor. To edit and extract a program, you need to know:

  1. The TiVo's IP address, or hostname. If you want to use a hostname, you should edit the /etc/hosts file and put the hostname and IP address in this file.
  2. The FSID list for the program. This consists of a set of FSID numbers separated by commas.
  3. The names of the two MPEG output files to want to save it as.
Now you can run the playitsam program:

% playitsam 1.2.3.4:643123,643137 outfile.mpa outfile.mpv
where 1.2.3.4 is the TiVo's IP address, 643123,643137 is the FSID list, and the other two arguments are the audio and video output files.

The program should open up an mpeg2dec window where you will see the video of the program. Note: mpeg2dec tries to play the video at the highest frame rate possible. This gives you very jerky video output, but this is tolerable if you are only trying to edit the program.

With the program now playing, you can edit it by typing these letters to Playitsam:

KeystrokeOperation
spaceToggle pause & playback
fGo forwards 1 chunk
1,2,3Go forwards 5, 25, 125 chunks
4,5,6Go forwards 500, 2500, 12500 chunks
bGo backwards 2 chunks
!,@,#Go backwards 5, 25, 125 chunks
$,%,Go backwards 500, 2500, 12500 chunks
)Go to the first chunk
cMake a cut starting at this chunk
eEnd the current cut at this chunk
sSet a splitpoint at this chunk
EEnd the current cut and also set a splitpoint
KKill all the cuts made so far
lList the cuts made so far, and size of edited recording
RGo into Record mode, and export the edited stream
q,Q,ctrl-CQuit Playitsam

Playitsam will only let you pause, start or end a cut at an MPEG `Group of Pictures' boundary. So editing isn't as fine-grained as you might like.

Once you have made your cuts and hit R, go have a coffee because at Ethernet speeds it will probably take a while.

6  Gplayitsam

Gplayitsam is the GTK version of Playitsam. It looks like this:

gpmain.gif

Note: the Step Forward and Step Backward buttons don't do anything. They are leftovers from the user interface design.

The scroll bar shows you where in the program you currently are, and the area to the right of the scroll bar shows you the current chunk number.

The Pause and Play buttons should be obvious. Note that the Pause button toggles pauses on/off, and the Play button acts as a Step button when you are in paused mode.

The buttons down the left-hand side work as per the command-line Playitsam, and you can also use the keystrokes `c', `e', `s', `E', `l', `K', `space' and `P' if you don't like clicking.

The File menu looks like this:

gpmenu.gif

Opening a stream from the TiVo gives you this dialog box:

gpopen.gif

Saving the edited program to separate MPEG streams gives you this dialog box:

gpsave.gif

Note that you only have to enter one filename. Gplayitsam will automatically append the .mpa and .mpv suffixes to the filename that you choose.

7  Multiplexing the MPEG Streams

The output from Playitsam is two MPEG elementary streams. The audio is 32kHz samplerate and the video probably has non-DVD or XSVCD dimensions. If you're going to write this out to DVD or XSVCD, you're probably going to have to resample both.

I have some scripts that use the Unix sox tool to upsample the audio, but I don't muck with the video stream.

I bought a specific DVD drive after I found out that the maximum stream bitrate for VCD and SVCD is around 3Mbps, and I am recording at High Quality which is around 4.5Mbps. So I found a DVD player which would play an XSVCD with a bitrate higher than the 3Mbps. I use mplex from the Siemens DVB toolset from the Siemens DVB toolset to multiplex them together to form an XSVCD stream.

% mplex -o output.mpg -t MPEG2 -i ES_STREAM outfile.mpa outfile.mpv
My DVD player can also play audio streams sampled at 32kHz, although I was also upsampling to 44.1kHz before I realised I didn't need to.

8  Optimising TiVo Network Transfers

You might find that the network transfers from the TiVo is slow. This is probably due to non-optimal setup of the TCP options on your Linux or FreeBSD box.

You may need to tune the TCP options on your Unix box so as to increase the TCP receive window and the TCP send window; this helps to keep the TCP stack transmitting and receiving data in the face of the inevitable LAN collisions. On my FreeBSD box and using a 2.5 TiVo, I used these sysctl commands at boot-time:

/sbin/sysctl -w net.inet.tcp.recvspace=65536
/sbin/sysctl -w net.inet.tcp.delayed_ack=0
/sbin/sysctl -w net.inet.tcp.sendspace=2400
These set the TCP receive window to 65,536 and the TCP send window to 131,072, and disable delayed TCP acknowledgements. However, I found that when I upgraded to a 2.5.1 TiVo system, I had to change the sendspace value from 131000 to 2400. In fact, the sendspace value seems to make a big difference! Going up to 3072, I could only get 0.1Mbps, but with 2400 I am getting 7Mbps over TiVoNet. If you are seeing very poor playback with Playitsam from the tyserver, then I would spend some time tuning your TCP configuration. I don't have a Linux box, so I can't tell you how to do the same tweaking on these systems. if you can supply details, I would be grateful.

9  Writing XSVCD Disks

I use vcdimager to create an XSVCD image from the MPEG file(s), and cdrdao to write the image out to a blank CD:

#!/bin/sh

vcdxgen -t svcd $*

vcdxbuild -c fred.cue -b fred.bin videocd.xml

cdrdao write -device 0,0,0 -speed 4 fred.cue

where $* is one or more output.mpg files.

10  Comments on the Source Code

The file NOTES contains my log of program development. For the most part, the code in lib is where the actions occurs.

mpeg.c is a bogus file, eventually it will have MPEG multiplexing code in it.

The way this version works is that we read in several chunks to build up a Group of Pictures, which includes a sequence header, the GOP header and then a series of I-, P- and B-frames. We then go back and find all the audio records which match the timestamps of the video frames.

Eventually we end up with a linked list which holds both the audio and video records. The linked list is described in new_tystream.h:

struct new_ty_node {

  u_int32_t type;      /* Type of entry */

  u_int32_t timestamp; /* Timestamp for the entry */

  u_int32_t size;      /* Size of data area in octets */

  u_int8_t *buf;       /* Pointer to the data area in memory */

  struct new_ty_node *next;   /* Pointer to next node in group of pictures */

};

and type is one of:

#define INVALID_ENTRY 0

#define GOP_ENTRY 1

#define SEQUENCE_ENTRY 2

#define I_ENTRY 3

#define P_ENTRY 4

#define B_ENTRY 5

#define AUDIO_ENTRY 6

Note: new_tystream.h also goes on about a new tystream format. I haven't written the code to deal with this format yet, so you can ignore this guff for now.

The heart of the extraction process is convert.c. build_group() takes a chunk number and builds a Group of Pictures linked list that starts at (or near) this chunk. We use parse_audchunks() to find the start and the end of this group, and we also build a linked list of audio data which belongs to this group.

We then use parse_vidchunk() to select the video data in the group. This comes in as raw chunk data, and we then massage this to be real MPEG records.

The final result is a struct new_ty_node linked list which is a single MPEG Group of Pictures. We also return the chunk number where the next group begins.

In terms of reading the code, you should read tools/nvsplit.c first to see how I use the split_stream() function. Then read split_stream() in lib/recplay.c to see where I use read_group(). You can ignore all the code dealing with cuts and editing and removal of B-frames for now.

read_group() in lib/readfile.c is really only a front-end for build_group(). I do this so I can eventually support input files which will already have Groups of Pictures.

Then you should read build_group() in convert.c

One thing to note is that the video data in the chunks is not contiguous; there are timestamps strewn amongst the video data. This makes it near impossible to correctly determine frame types etc. Therefore, each time I read in video data, I memcpy() it into a separate contiguous buffer called VidBuf. Then I point the nodes in the Video linked list into this buffer and not the original chunk buffers. Yes it means more memory use but I can't think of another way of doing it.

11  Contact Details

If you have any questions or comments, please e-mail me at wkt@tuhs.org. However, remember that this is a proof of concept version; I will readily accept patchs and bug fixes. I probably won't do much else.

Warren Toomey, January 2003.


File translated from TEX by TTH, version 2.78.
On 10 Jan 2003, 09:38.