Quantcast
Channel: Active questions tagged rest - Stack Overflow
Viewing all articles
Browse latest Browse all 3667

How should data be processed when returned from a Trino database?

$
0
0

I am attempting to fetch data from a Trino database. I send my initial query to the database and get a NextURI. In a while loop after that, I re-check the NextURI to get part of the data until the Trino connection finishes sending overall data.

The issue is that I get no data and endless NextURI's.

The code below reflects this process. Note that it starts off by getting an authentication token for Trino before continuing.

app.get("/getOpenskyData", async (req: Request, res: Response) => {  var startTime = new Date().getTime();  console.log("Server | getHistoricalADSBData | Called");  try {    const startTimeEPOCHMS = Number(req.query.startTimeEPOCHMS);    const endTimeEPOCHMS = Number(req.query.endTimeEPOCHMS);    interface TokenPayload {      access_token: string;      exp: number;    }    // Get the token from OpenSky Network    const result = await axios.post("https://auth.opensky-network.org/auth/realms/opensky-network/protocol/openid-connect/token",      new URLSearchParams({        client_id: "trino-client",        grant_type: "password",        username: process.env.OPENSKY_USERNAME +"",        password: process.env.OPENSKY_PASSWORD +""      }).toString(),      {        headers: {'Content-Type': 'application/x-www-form-urlencoded'        }      }    );    const payload = result.data;    const decoded: TokenPayload = jwtDecode(payload.access_token);    // Ensure the token has an exp property    if (!decoded.exp) {      throw new Error("Token does not have an expiration time");    }    console.log("Server | getHistoricalADSBData | Got Token");    const queryString = `SELECT * FROM minio.osky.state_vectors_data4 WHERE time BETWEEN ${startTimeEPOCHMS / 1000} AND ${endTimeEPOCHMS / 1000}`;    // Execute the query on Trino    const response = await axios.post('https://trino.opensky-network.org/v1/statement',      queryString,      {        headers: {          Authorization: `Bearer ${payload.access_token}`,'Content-Type': 'text/plain','X-Trino-Buffer': 'true',        },      }    );    if (response && response.status === 200) {      const body = response.data;      const results: any[] = [];      let nextUri = body.nextUri; // Initialize nextUri      // Loop to handle pagination      while (nextUri) {        const fetchResponse = await axios.get(nextUri, {          headers: {            Authorization: `Bearer ${payload.access_token}`,          },        });        if (fetchResponse && fetchResponse.status === 200) {          const fetchBody = fetchResponse.data;          if (fetchBody.data && fetchBody.data.length > 0) {            results.push(...fetchBody.columns); // Push data to results array          }          console.log(fetchBody.columns)          nextUri = fetchBody.nextUri; // Update nextUri          console.log(fetchBody.nextUri)        } else {          throw new Error('Unokay response whilst fetching data from OpenSky');        }      }      console.log("Server | getHistoricalADSBData | Results:", results);      res.status(200).json(results);    } else {      throw new Error('Unokay response whilst querying OpenSky');    }    console.log("Server | getHistoricalADSBData | Took " + (new Date().getTime() - startTime) +" ms");  } catch (error) {    console.error('Server | getHistoricalADSBData | Error of: ', (error as unknown as Error).message);    return res.status(500).json({ message: 'Internal server error while fetching historical ADSB.' });  }});

When I print the fetchResponse.columns field, I get the following results. Note that it is missing all of the required data, only including an array of a single ICAO24 value. I should be getting 100's of thousands of pieces of data for this request, as it is getting global ADSB data for one second.

[  {    name: 'time',    type: 'integer',    typeSignature: { rawType: 'integer', arguments: [] }  },  {    name: 'icao24',    type: 'varchar',    typeSignature: { rawType: 'varchar', arguments: [Array] }  },  {    name: 'lat',    type: 'double',    typeSignature: { rawType: 'double', arguments: [] }  },  {    name: 'lon',    type: 'double',    typeSignature: { rawType: 'double', arguments: [] }  },  {    name: 'velocity',    type: 'double',    typeSignature: { rawType: 'double', arguments: [] }  },  {    name: 'heading',    type: 'double',    typeSignature: { rawType: 'double', arguments: [] }  },  {    name: 'vertrate',    type: 'double',    typeSignature: { rawType: 'double', arguments: [] }  },  {    name: 'callsign',    type: 'varchar',    typeSignature: { rawType: 'varchar', arguments: [Array] }  },  {    name: 'onground',    type: 'boolean',    typeSignature: { rawType: 'boolean', arguments: [] }  },  {    name: 'alert',    type: 'boolean',    typeSignature: { rawType: 'boolean', arguments: [] }  },  {    name: 'spi',    type: 'boolean',    typeSignature: { rawType: 'boolean', arguments: [] }  },  {    name: 'squawk',    type: 'varchar',    typeSignature: { rawType: 'varchar', arguments: [Array] }  },  {    name: 'baroaltitude',    type: 'double',    typeSignature: { rawType: 'double', arguments: [] }  },  {    name: 'geoaltitude',    type: 'double',    typeSignature: { rawType: 'double', arguments: [] }  },  {    name: 'lastposupdate',    type: 'double',    typeSignature: { rawType: 'double', arguments: [] }  },  {    name: 'lastcontact',    type: 'double',    typeSignature: { rawType: 'double', arguments: [] }  },  {    name: 'serials',    type: 'array(integer)',    typeSignature: { rawType: 'array', arguments: [Array] }  },  {    name: 'hour',    type: 'integer',    typeSignature: { rawType: 'integer', arguments: [] }  }]

The nextURI looks like the following:

https://trino.opensky-network.org/v1/statement/executing/20240619_210740_88003_jtdt5/y2e8bb6c86212fff8d50ab161a4300917292bc401/278

Note that I am 278 NextURIs in (so 278 seconds of processing) and am not getting data.

I can't find any documentation online on how Trino is supposed to format data when it sends it back, and I can't find any documentation on how to parse that data in JavaScript.

The query should not take 278 seconds to complete or return no data.

How am I supposed to process Trino data in Typescript correctly?


Viewing all articles
Browse latest Browse all 3667

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>