I am attempting to fetch data from a Trino database. I send my initial query to the database and get a NextURI. In a while loop after that, I re-check the NextURI to get part of the data until the Trino connection finishes sending overall data.
The issue is that I get no data and endless NextURI's.
The code below reflects this process. Note that it starts off by getting an authentication token for Trino before continuing.
app.get("/getOpenskyData", async (req: Request, res: Response) => { var startTime = new Date().getTime(); console.log("Server | getHistoricalADSBData | Called"); try { const startTimeEPOCHMS = Number(req.query.startTimeEPOCHMS); const endTimeEPOCHMS = Number(req.query.endTimeEPOCHMS); interface TokenPayload { access_token: string; exp: number; } // Get the token from OpenSky Network const result = await axios.post("https://auth.opensky-network.org/auth/realms/opensky-network/protocol/openid-connect/token", new URLSearchParams({ client_id: "trino-client", grant_type: "password", username: process.env.OPENSKY_USERNAME +"", password: process.env.OPENSKY_PASSWORD +"" }).toString(), { headers: {'Content-Type': 'application/x-www-form-urlencoded' } } ); const payload = result.data; const decoded: TokenPayload = jwtDecode(payload.access_token); // Ensure the token has an exp property if (!decoded.exp) { throw new Error("Token does not have an expiration time"); } console.log("Server | getHistoricalADSBData | Got Token"); const queryString = `SELECT * FROM minio.osky.state_vectors_data4 WHERE time BETWEEN ${startTimeEPOCHMS / 1000} AND ${endTimeEPOCHMS / 1000}`; // Execute the query on Trino const response = await axios.post('https://trino.opensky-network.org/v1/statement', queryString, { headers: { Authorization: `Bearer ${payload.access_token}`,'Content-Type': 'text/plain','X-Trino-Buffer': 'true', }, } ); if (response && response.status === 200) { const body = response.data; const results: any[] = []; let nextUri = body.nextUri; // Initialize nextUri // Loop to handle pagination while (nextUri) { const fetchResponse = await axios.get(nextUri, { headers: { Authorization: `Bearer ${payload.access_token}`, }, }); if (fetchResponse && fetchResponse.status === 200) { const fetchBody = fetchResponse.data; if (fetchBody.data && fetchBody.data.length > 0) { results.push(...fetchBody.columns); // Push data to results array } console.log(fetchBody.columns) nextUri = fetchBody.nextUri; // Update nextUri console.log(fetchBody.nextUri) } else { throw new Error('Unokay response whilst fetching data from OpenSky'); } } console.log("Server | getHistoricalADSBData | Results:", results); res.status(200).json(results); } else { throw new Error('Unokay response whilst querying OpenSky'); } console.log("Server | getHistoricalADSBData | Took " + (new Date().getTime() - startTime) +" ms"); } catch (error) { console.error('Server | getHistoricalADSBData | Error of: ', (error as unknown as Error).message); return res.status(500).json({ message: 'Internal server error while fetching historical ADSB.' }); }});
When I print the fetchResponse.columns
field, I get the following results. Note that it is missing all of the required data, only including an array of a single ICAO24 value. I should be getting 100's of thousands of pieces of data for this request, as it is getting global ADSB data for one second.
[ { name: 'time', type: 'integer', typeSignature: { rawType: 'integer', arguments: [] } }, { name: 'icao24', type: 'varchar', typeSignature: { rawType: 'varchar', arguments: [Array] } }, { name: 'lat', type: 'double', typeSignature: { rawType: 'double', arguments: [] } }, { name: 'lon', type: 'double', typeSignature: { rawType: 'double', arguments: [] } }, { name: 'velocity', type: 'double', typeSignature: { rawType: 'double', arguments: [] } }, { name: 'heading', type: 'double', typeSignature: { rawType: 'double', arguments: [] } }, { name: 'vertrate', type: 'double', typeSignature: { rawType: 'double', arguments: [] } }, { name: 'callsign', type: 'varchar', typeSignature: { rawType: 'varchar', arguments: [Array] } }, { name: 'onground', type: 'boolean', typeSignature: { rawType: 'boolean', arguments: [] } }, { name: 'alert', type: 'boolean', typeSignature: { rawType: 'boolean', arguments: [] } }, { name: 'spi', type: 'boolean', typeSignature: { rawType: 'boolean', arguments: [] } }, { name: 'squawk', type: 'varchar', typeSignature: { rawType: 'varchar', arguments: [Array] } }, { name: 'baroaltitude', type: 'double', typeSignature: { rawType: 'double', arguments: [] } }, { name: 'geoaltitude', type: 'double', typeSignature: { rawType: 'double', arguments: [] } }, { name: 'lastposupdate', type: 'double', typeSignature: { rawType: 'double', arguments: [] } }, { name: 'lastcontact', type: 'double', typeSignature: { rawType: 'double', arguments: [] } }, { name: 'serials', type: 'array(integer)', typeSignature: { rawType: 'array', arguments: [Array] } }, { name: 'hour', type: 'integer', typeSignature: { rawType: 'integer', arguments: [] } }]
The nextURI looks like the following:
https://trino.opensky-network.org/v1/statement/executing/20240619_210740_88003_jtdt5/y2e8bb6c86212fff8d50ab161a4300917292bc401/278
Note that I am 278 NextURIs in (so 278 seconds of processing) and am not getting data.
I can't find any documentation online on how Trino is supposed to format data when it sends it back, and I can't find any documentation on how to parse that data in JavaScript.
The query should not take 278 seconds to complete or return no data.
How am I supposed to process Trino data in Typescript correctly?