FTP Facilities at CDS: Getting compressed, zipped, tared,
and FITS data


Available transformations in an interactive FTP session at CDS (cdsarc.cds.unistra.fr)
part of a fileuncompressed fileUnix compressedgziptarzipFITS

The examples given here assume a FTP connection in command mode – they work also from a browser like Netscape or Explorer using a Location starting by ftp://cdsarc.cds.unistra.fr/pub/cats/

1  File Compression and Decompression

The compressed versions of files are generally between 3 and 5 times smaller than the uncompressed version. When the network is slow, it's therefore recommended to ask for compressed versions of files.

Our FTP server knows about compression/decompression, in one of the classical compression formats:

In the CDS archive, files larger than 1Mbyte are sometimes compressed, but some large catalogs which are directly accessed by query programs are kept uncompressed. Files compressed with the compress standard Unix program have a name terminated with the suffix .Z (uppercase), and files compressed by gzip utility have a suffix .gz (lowercase). The existence of one of these suffixes is the only indication that the file is compressed.

Large files which are not compressed can be compressed ``on the fly'' with the addition of the uppercase .Z suffix (Unix compress), the lowercase .gz suffix (gzip function), or the lowercase .zip suffix (for PCs). For instance, getting a compressed version of file table6.dat is achieved with the command

ftp> get table6.dat.gz

which creates on your computer the file table6.dat.gz. Remember that you can rename a file in the second argument of the get command, i.e.

ftp> get table6.dat.gz results.gz

creates on your computer a file named results.gz instead of the default table6.dat.gz .

Files can similarly be uncompressed by just omitting the suffix .gz or .Z in the get command of the ftp program.

2   Transformation to FITS

FITS (Flexible Image Transport Sytem) is a format widely used in astronomy for transporting CCD or radioastronomical images; it also includes an extension for the description of data in tabular form which is also known by the major astronomical data analysis systems.

The Standard for Astronomical Catalogues used for the description of astronomical catalogues was partly designed for the purpose of allowing an automatic conversion of the tables stored as plain ascii files, into other formats, and the ReadMe file containing the descriptions of the tabular material is interpreted by our FTP server and the translation ``on the fly'' into FITS files is now possible: the remote user has simply to add of a .fit extension to the name of the file to be retrieved. For instance, the FITS version of table6.dat file can be copied with the command

ftp> get table6.fit

(Note that the .dat was omitted; the command get table6.dat.fit would also have worked, but the suffix .dat generally indicates data tables which transformation into FITS is meaningful)

The FITS version can be further compressed, either with the Unix standard compress (append the suffix .fit.Z to the file name), or the gzip compression (append the suffix .fit.gz or .fgz to the file name).

All data files in a directory can also be combined into a single FITS file.

3  Copying full directories

All files stored un a directory can be combined into a single file, in the tar format (Unix standard), the zip format (for PCs), or in FITS format. The result can further be compressed:

Remind that the dot (.) indicates the current directory; this means that on your PC, when you are located in the directory corresponding to what you are insterested in, you can get a copy of all files with the single command

ftp> get ..zip thefiles.zip

which generates on your PC the file thefiles.zip, which can be converted to the original files with the unzip or winzip command.

Similarly, all tables making up a catalogue may also be retrieved as a single FITS file: the command

ftp> get ..fgz tables.fgz

retrieves as a single compressed FITS file named tables.fgz all tabular material stored in the current directory.

4  Partial File Copies

It is possible to get an extract of a file; this is useful e.g. when the network is slow, and in a previous FTP session you just missed the end of a file. You can start a copy to a given location with the colon (:), followed the starting point, eventually a dash (-) and the ending point, and eventually the unit of copy as the letter b, c, k or l for blocks (1 block = 512 bytes), characters (1 character = 1byte), kbytes (1Kb = 1024 bytes); the default is a line.

For instance:

ftp> get bigfile.dat:5000 bigfile.nxt

will copy the file bigfile.dat from line 5000 on into a file named bigfile.nxt on your computer.

For a binary file, if you already copied 123456 bytes of the file image.fit, you can copy the remaining bytes as the file image.nxt with the command:

ftp> get image.fit:123457c image.nxt

5  Access to the Astronomical Catalogues at CDS via FTP

An anonymous account for copying astronomical catalogues has been existing at CDS node cdsarc.cds.unistra.fr since 1992, and is the most frequently used way for copying catalogues at CDS.

The way the astronomical catalogues are organized and described at CDS and in other major astronomical data centres is described in the Standard for Astronomical Catalogues.

The navigation among the various catalogues and tables has been simplified: moving to the directory corresponding to a catalogue or publication can be performed from anywhere. For instance, after having visited the VI/65 catalogue, typing

ftp> cd A+AS/108/669

will change the current directory to /pub/cats/J/A+AS/108/669 which corresponds to the article by Niedzielski et al. published in A&ASup. 108, 669 (1994A&AS..108..669N). The server also understands the catalogue numbering used at the ADC (Astronomical Data Center at Goddard Space Flight Center, NASA): typing

ftp> cd 6065

has the same effect as typing

ftp> cd VI/65.

Similarly, typing

ftp> cd a&as/108/669

has the same effect as typing

ftp> cd J/A+AS/108/669.

A message informs about the contents of the current directory when moving across the file system; for instance, the following message appears after moving to directory /pub/cats/VI/65 corresponding to catalogue VI/65:

==========================================================================
VI/65   ULTRAVIOLET RADIATION FROM EVOLVED STELLAR POPULATIONS. I. Models
==========================================================================


The CDS ftpd server is based on the wu-ftpd code from the Washington University Data Archive in St Louis.

Last Update: 10 May 2001      François Ochsenbein ()


CDS · Simbad · VizieR · Aladin · Catalogues · Nomenclature · Biblio · Tutorial · Developer's corner Question@simbad