Astronomical Data Analysis Software and Systems XVI P2.01 ASP Conference Series, Vol. XXX, 2006 R. Shaw, F. Hill and D. Bell, eds. A Customizable Database Server: MCS, a Flexible Resource for Astronomical Projects Luciano Nicastro INAF–IASF, Via P. Gobetti 101, 40129 Bologna, Italy 7 0 Giorgio Calderone 0 2 INAF–IASF, Via U. La Malfa 153, 90146 Palermo, Italy n a Abstract. Nowadays medium-large size astronomical projects have to J face the management of a large amount of information and data. Ded- 9 icated data centres manage the collection of raw and processed data 1 and consequently make them accessible, typically as static files, either v via (s)ftp or http (web). However the various steps of data acquisition, 0 archiving, processing and delivering are accomplished by different tools. 6 The data production information (logging) is not always collected into 2 relational databases tables causing long delays before people know about 1 0 their existence. Luckily the use, in many cases, of standard file formats 7 like FITS can help to track the data origin and processing status. As 0 Virtual Observatory standards will become more common, things will / h quickly improve. Here we present a software library we implemented in p order to usea unifiedmodelin astronomical datatreatment. All thedata - o are stored into the same database becoming available in different forms r to different users with different privileges. t s a : v i X 1. Introduction r a Information services can be separated in two classes: those in which the infor- mation produced is addressed to humans, and those in which it is meant to be used by other software applications. In the former case there is a quite stan- dardized way to develop such an information service, essentially based upon a web server, a database (DB) server, a scripting language and HTML pages. In the latter case instead there is no such standardization, and here’s why MCS (My Customizable Server) was implemented (Calderone & Nicastro 2006). MCS is a set of software tools aimed at easily implementing information services, that is an application providing a service over the network. At the core of an MCS-based system there is a TCP server listening for user connections requests. When a user is connected, the server will send him all the requested information. However the transmitted data aren’t in free format (like in a web page), but are packed in a well defined fashion (using the MCS protocol) so that on the other side a software can understand what is being sent. The MCS high level classes will hide all code implementations related to multi-threading, networking,DBaccess,etc.,andrequirenolow-levelknowledgeoftheseissuesby 1 2 Nicastro & Calderone the users. MCS, as its name suggests, can be customized through the derivation of some classes. So MCS and its protocol are for software applications what a web server and HTTP are for the WWW: a simple way to access any sort of data. In this comparison customizing the MCS server is like writing a web page. The usage of MCS for the automatic management of the data collected by the robotic telescope REM is described in Nicastro & Calderone (2006). MCSisdeveloped ontheGNU/Linux platform anditisan ongoingproject. Itis released undertheGPL license and can befreely downloaded from the web. The site contains all news, updates, documentation and downloadable software packages. 2. The MCS Architecture and Components The main features of application servers built with MCS are: easy to configure, authentication and grant support,secure connections (through SSL),files trans- fer, DB access (MySQL), base commands set, supportto create new customized commands, accessibility (as client) fromother languages (C,Fortran, IDL,PHP, Python, etc.), XML,FITSandVOTAble management, HTMandHEALPix sky pixelization support, logging facility, etc. All these features are already available, without performing any customization, and more will beadded in the future. So to implement a simple service with the above features you’ll only need to install MCS and configure it through a simple configuration file. The only code needed for the “simplest” server is as follows: #include <mcs.hh> //Mandatory includes for all using namespace mcs; // MCS-based applications int main(int argc, char *argv[]) { //Main program Env* env = mcsStart("simplest"); //Start the server... mcsWait(env); } //...and wait for its end Formorecomplexservicesyoucancustomizetheserverbehaviorinseveralways: • adding external programs, either real external applications or batch lists of MCS commands; • adding SQL programs, to be executed on the DB server; • adding customized commands, deriving the UserThread class; • modifyingthebehavioroftheserversidethread,derivingtheLocalThread class; In a typical application you should implement a DB with all the tables needed to store the relevant data related to a project, eventually preparing the required external programs. A typical architecture of an MCS-based system is: Database server: the DB server is used to handle clients authentication, to store all application specific data and anything else necessary to the applica- tion itself. This server isn’t accessible directly from the clients, but it is visible only to the application server. At the moment the only supported DB server is MySQL. Other servers could become accessible in the future. Application server: the application server is the core of the information sys- tem. Itimplementstheclient/servermodel: aclientopensaTCPsockettowards the host running MCS and sends a request, then the server “computes” an an- swer,eventually queryingtheDBand/orexecutingsomeexternalprograms,and A Customizable Database Server 3 sends it back to the client. The behavior of the MCS server can be customized. External programs: external programs are software applications, written in any language, which interact with the application server via command line and the standard output. Support to these programs was added to easily integrate in MCS already existing applications. Clients: clients are programs which access the MCS service over the network. Such programs can be written in any language and run on any platform, pro- vided that they implement the MCS protocol. Interfaces that implement the MCS protocol are provided by the MCS library for the following languages on the Linux platform: C++, C, Fortran, IDL, PHP, Python. Support for other languages, such as Java, Perl and Tcl/Tk, and the Windows platform will be available soon. MCS-based systems are “open”, in the sense that existing DB and software tools can be easily integrated. Moreover MCS doesn’t make any assumption about type and quantity of data you need to deal with. From a user’s point of view MCS is very similar to the usage of a classic Unix shell, that is a command line interface with a prompt on which users can execute commands in their own environment and wait for the output before a new command is issued. It is therefore possible to make a comparison between the “components” of a shell, and those of an MCS connection: Unix shell MCS server stdin and stdout → bidirectional TCP socket system account → MySQL account internal commands → base commands programs, shell scripts → external programs (EXEC command) home directory → work directory Sent/received data are binary formatted using the MCS protocol and then the output of a command won’t be ASCII text like in a shell. Deriving a C++ class means creating a new class that preserves all the characteristics of the parent class but with more specific behaviors added. In the MCS case, the server behavior can be customized through the derivation of the UserThread class (as described in the documentation, only the “virtual” methods should be overloaded). This way it is possible to create custom com- mands available as if they were “base commands”. Another class that can be derived for customization is the LocalThread. It runs in a server side thread, independently from other client threads. This class can be used to implement some server side tasks, like data quick look or reduction, DB maintenance, etc. 3. Other Uses of the MCS Library and the Companion Tools The MCS library contains several classes to deal with different tasks. These classes are mainly used by the MCS server itself, but can also be used without a running MCS server. Some of the tasks covered by these classes are: threads and synchronization, sockets, parsing of command lines, DB access, VOTable and FITS read/write. Itispossibletoexecutequeriesona(MySQL)DBserverusingtheDBConn, Query, Record and Table classes. All these classes can be used not only with 4 Nicastro & Calderone C++ programs, but also with all those programming languages for which we have an interface: C, Fortran, IDL, PHP, Python (Java, Perl and Tcl/Tk will be available soon). The VOT Parser Table class provides an easy interface to read VOTable and FITS files at the highest level. At lower level VOT Parser Stream can be used to read a stream (like a SAX parser) and VOT Parser Tree to read a tree in memory (like a DOM parser). Of course, also these classes are available to languages for which we have an interface. For all the details please see the documentation on the web site! MCS is accompanied by a number of software tools to be used with the MySQL DB server to improve its functionalities. These tools, once installed, become part of the DB server itself and the facilities they provide are immedi- ately and transparently available to anyone who connects to that server with no need to write a single line of code: MyRO (My Record Oriented privilege system): it provides a natural exten- sion to the MySQL privilege system, offering the possibility to specify privileges (user/group) on a record level (requires MySQL ≥ 5.0); DBEngine for FITSand VOTable: itgives MySQLtheability toread/write DB tables in VOTable and FITS format, which means that you can perform queries on them, as well as write into them as a result of a computation from other tables (under development); Database indexing using HEALPix and HTM: this software provides in- dexingfacilityfordatawithsphericalcoordinatesallowing,e.g.,veryfastcircular and rectangular selections of entries in very large tables (under development). 4. Conclusions and the Future We believe MCS is an extremely usefultool and it is in continuous development. It will soon implement interfaces for other languages such as Perl, Java, Tcl/Tk. Another feature that will be implemented soon is the integration with MyRO to handle a more complex privilege system. User contributed libraries for the varioussupportedlanguagesarebeingbuilt. Thiswillallowaneveneasieraccess to the MCS functionalities to non expert programmers. Moreover, commonly used, independentlydeveloped external packages willalso beincluded and made accessible trough MCS. They include the afore mentioned HEALPix and HTM libraries for sky pixelization scheme, the Naval Observatory Vector Astrometry Subroutines (NOVAS) used for astrometric calculations and transformations, the World Coordinate System (WCS) library and tools. References Calderone, G. & Nicastro, L. 2006, in Neutron Stars and Pulsars, MPE-Report No. 291, astro-ph/0701102 Nicastro, L. & Calderone, G. 2006, in Neutron Stars and Pulsars, MPE-Report No. 291, astro-ph/0701099