Network Data Collector
Node.js Web App

As the lead developer, I developed this Web App using HTML, CSS, and Javascript for the frontend and Node.js and Express for the backend.

Description

Previously, I developed a Google Chrome Extension to automate data collection of a specific webpage. I was able to accomplish this by developing the Chrome Extension into a web scraper and exporting this data locally onto a Excel .csv file. The goal of this project was to build off the extension and find a way to export the data externally through other means. With all options considered, I decided to use Node.js and develop a Full Stack application. For this part of the web application, I chose to use the Nodemon, Google API, and Puppeteer libraries to support the application. Nodemon created more accessibility for the project, especially for readily updating and testing new code. Through Google Cloud, I was able to create a Google service account to access Google's API services. With this access, I was able to create new folders and Google Sheets, append new data to these spreadsheets, and create permissions for these new files and folders. Lastly, Puppeteer allowed me to open a browser in the backend out of the view of the user, in which was an issue regarding the Chrome Extension. With the browser in the backend, I was easily able to manipulate the webpage and access the necessary HTML elements containing the information I needed.

In addition to creating and appending to Google Sheets, this part of the Web App also has the following features: running ping and iPerf throughput tests, automatically create graphs in Google Sheets based on the data collected from the spreadsheet, send an automatic email of the spreadsheets and graphs as a pdf. The ping and throughput testing is done on a timed basis, in which the tests run individually on a timer set by the user in a configuration file.

With this Web App, there are currently four different functions: Network Data Collection (Long-term Collection), Single Data Collection (Short-Term Collection + Live User Display), Online Mapping, and Offline Post Processing.

Languages

Frontend: HTML, CSS, Javascript

Backend: Node.js, Express