{% extends 'app_base.html' %} {% load static %} {% if user.is_authenticated %} {% block content_wrapper %}

FAQs

Building Block Description

FAQs

This section of the dashboard helps the users to get an overview of the existing FAQs and their answers.


IT SECURITY

1. How will the company data be accessed on premise?

  • EcoKI offers three options for accessing your company's data:

2. How will the company data be accessed remotely?

  • The on-premise central database, the on-premise running ecoKI building blocks and pipelines and the central database need to be accessible remotely by the application engineers.
  • This is necessary for the development, configuration and maintenance of our ecoKI-solutions.
  • The connection will be established via a secure VPN-connection to the server(s), where the following services run:
    • Central database
    • Running ecoKI building blocks and pipelines

3. How will the company data be stored off-premise?

  • For funded research projects, data will need to be stored on the servers of the partner universities. This is necessary for the researchers, so they can work with the data.
  • A secure VPN connection will be used to transfer the data to the off-premise servers.
  • The full data will be stored at one location. Project partners will only be able to download limited amounts of the data from this location (details to be agreed upon in project). Full access to data will only be possible via secure VPN connection.

4. What is expected from the company in terms of data security?

    Answers to the following questions are expected from your company, regarding data security:

  • Understanding the IT infrastructure and security measures on the SME side
    • What kind of connection options are possible ?
    • How is data stored (for example on the plant side or in the cloud etc). What kind of measures are required to access the data ?
    • Which virus protection software is used and what needs are to be considered to when interacting with it ?
  • Getting the access/connection rights
    • What is the procedure for online connection ?
    • What is the procedure for offline connection ?
  • Understanding the data policy of the SME
    • Is it allowed to store the data in off-premise ecoKI central database ?
    • How can we use the data and publish results from the project (model development, successs stories etc.) ?
    • What kind of actions or documentation does SME expect from the ecoKI team regarding data security ?

5. How will transparency be guaranteed on where which data is used?

  • Q1 through Q3 explain where data may be used in projects done using ecoKI. To guarantee transparency on this individual projects, a legally binding confidentiality agreement will be signed by all participants, clearly stating where (e.g. on/off premise, etc.) which data will be stored and used.
  • Confidentiality agreement will include:
    • On which servers data will be stored
    • Which institutions will have access to data
    • How data will be accessed

6. How will transparency be guaranteed for what the data is used for?

    ecoKI ensures transparency regarding what the data is used for by following the key practices lister below:

  • Involve the client in the process
    • A close collaboaration is established between your company and the Application Engineer (AE) at every step of the project, including the requirements analaysis step, where your company can define its data usage constraints. Additionally, the AE is required to present a detailed work plan that includes the intended use of data at any point in the project
  • Establish internal data usage transparency policy
    • Your company's requirements on the intended use of the datacan be expressed as an internal data usage transparency policy that both your company and the AE have to explicitly agree upon. For example, your company may want to make sure that no personal data, such as people's faces in images is used in the project.
  • Collect necessary data only
    • By doing so, leeping track of what the data is used for becomes easier
  • Code and dataset transparency
    • To guarantee full transparency on how your company's data is used, the source code of all ecoKI's building blocks as well as the project-specific code developed by the AE, are available to your company. Additionally, data changed during the course of the project (e.g. cleansed or transformed data) is also made available to your company.

7. How will IT security be guaranteed when using a cloud approach ?

To be answered

9. How will viruses be avoided when new data is collected with edge devices ?

To be answered

DATA MODELLING

1. What is a data-model?

  • Generally speaking, a data model is a conceptual representation of the data structures and relationships that exist within a particular domain or application. It provides a way to organize and structure data so that it can be easily managed, queried, and analyzed. Data models are used by software developers, database administrators, and data analysts to design and implement databases, data warehouses, and other data-related systems. They help to ensure that data is stored and accessed in a consistent and efficient manner, and they provide a foundation for data analysis and reporting.
  • https://drive.google.com/file/d/1Q4BmFkgl3LSo_6rZ0bWgzlfF8MNAn1fN/view?usp=sharing
  • Data Model

2. What is the importance of data-model?

  • A data model provides a visual representation of the data, its structure, and its relationships, which makes it easier for developers, analysts, and stakeholders to understand and discuss the data. It ensures that data is accurate and consistent across the entire system. This helps to prevent errors, inconsistencies, and redundancies that can occur when data is stored and managed in an ad-hoc manner. It can also help to optimize the performance and efficiency of the system by defining the most efficient way to store and access the data. By minimizing the number of database queries and reducing redundant data, a data model can help to improve the overall performance and responsiveness of the system.

3. Which Database is used in ecoKI platform? Is it different from the other conventional database?

  • MongoDB is used for data-storage in ecoKI. It is a source-available cross-platform document-oriented database program. Classified as a NoSQL database program, MongoDB uses JSON-like documents with optional schemas. Instead of tables, a MongoDB database stores its data in collections. A collection holds one or more BSON documents. Documents are analogous to records or rows in a relational database table. Each document has one or more fields; fields are similar to the columns in a relational database table. A record in MongoDB is a document, which is a data structure composed of field and value pairs. It's important to note that the suitability of MongoDB or a SQL database depends on specific use cases, requirements, and preferences. While MongoDB excels in certain areas such as scalability and flexibility. Below is a table outlining the pros and cons of MongoDB compared to a traditional SQL database:
  • Aspect MongoDB SQL Database
    Data Model Document-oriented data model Table-based data model
    Schema Flexibility Flexible schema allows for dynamic data structure Strict schema requires predefined structure
    Scalability Horizontal scaling with sharding Vertical scaling with replication and clustering
    Performance High performance for read-heavy workloads Performance can degrade with complex queries
    Query Language Rich query language with MongoDB Query Language (MQL) SQL for querying data
    Data Integrity Eventual consistency model ACID compliance ensures data integrity
    Complex Transactions Limited support for multi-document transactions Full support for complex transactions
    Relationships No native support for JOIN operations Supports complex JOIN operations
    Development Flexibility Well-suited for agile development and rapid iteration Traditional development requires careful schema design
    Community & Support Active community and extensive documentation Established ecosystem with wide support

4. How can I get connected to central database?

  • With the IP address, you can get connected to the central database. The building block ecoki_data_reader provides the functionality to connect to a MongoDB server with a specific IP address. After establishing the connection to the database server, users can continue selecting database and its collection in drop-down menus.

5. How do I bring the data into the format of the data model and upload the company's data?

  • Converting tabular data into data model requires a json configuration file mapping the columns of the tabular data to different properties of the data model. Company’s data can be uploaded into MongoDB by using the building block data_writer. In this building block users have to upload this json configuration file and configure the database settings, like IP address, database and collection. .

6. Do I have to upload my data into the central ecoKI database (MongoDB)? If yes, does it have to be in JSON format? If yes, does it have to be in the data model format? What is the recommendation?

  • No, you should upload in the local MongoDB on your own system. The centralized database is restricted only for the development purposes of the ecoKI Team. For configuration refer to the question no:5. Recommendation, is Yes, using the right data model is essential for effectively storing unstructured data from process plants and manufacturing factories. It enables efficient storage, ensures data integrity, facilitates data retrieval and analysis, integrates with analytics tools, and provides scalability and flexibility to accommodate evolving data requirements.
PIPELINE

1. How a pipeline can be configured

  • Using settings.json located in the corresponding pipeline folder.

2. What steps are needed to run the pipeline?

  • 1. create the pipeline on the pipeline overview page of the central dashboard. 2. open the created pipeline in the active pipelines and run it by clicking on run pipeline or configure run pipeline button.

3. What need to be done to send the data from one building block to another w.r.t pipeline settings.json file? what configuration are needed?

  • The connection attribute in settings.json defines the connection between building blocks. Users have to specify the name of the source and target building block and also the name of their output and input ports.

4. How can the parameters of a pipeline can be modified?

  • The parameter of custom pipelines can be modified via settings.json. ecoKI pipelines can not be changed or overwritten. Hence, users have to save the modified ecoKI pipelines as a new custom pipeline.

5. How do I use buildings pipelines in combination with my own code? E.g. Do my own pre-processing, but then use the modelling BB, or vice versa.

  • Define input and output ports for a new building block and implement the execute method by the new code of users.

6. I am new to ecoKI. Is there an example pipeline illustrating the basic concepts that I can run quickly?

  • Examples of some pipelines are present in “Custom Pipelines” category on the dashboard.

7. How to run the pipeline on windows?

  • The same as on other operating systems.

8. How can I create a new pipeline?

  • Please, read the answers to above questions.
BUILDING BLOCK

1. How can the parameters of a building block can be modified?

  • The parameters of a building block can be set via the settings.json file where the pipeline using said building block is defined. This is done via the settings field, which consists in a dictionary whose keys (resp. values) represent the building block’s parameter names (resp. values). The mapping between the values in the settings.json file and the building block parameters is defined in the execute() method of the building block.

2. How do I run building blocks?

  • Building blocks are standalone python classes with a singular purpose. Running or testing an individual building block is an advanced skill and in order do it, one needs to first understand what the purpose of the particular building block is, for this look in the READ.ME to get an overview of the functionality, the inputs and outputs etc. There are two ways to test run a building block (subject to feasibility). a. Method-1: Using ecoki architecture: i. Create a custom pipeline including the building block with an appropriate settings.json file. (refer to pipeline FAQs) ii. Subject to the needs, one could print out the results on the console or create a custom visualizer building block to view the results on the dashboard. iii. Refer to ecoki/examples/energy_monitoring/energyMonitoring.py to see an example of how a building block is added to a settings.json, with its own visualization module. iv. Execute the pipeline via Dashboard and view the results on console or the dashboard b. Method-2: Python i. A building class could be instantiated and executed locally as a python module using the LocalBuildingBlockExecutor feature of the EcoKI architecture. ii. Refer to the following example for a detailed view ecoki/examples/execute_eocki_pipeline/ecoKIBBExectuorExamples.ipynb

3. How do I use buildings blocks in combination with my own code? E.g. Do my own pre-processing, but then use the modelling BB, or vice versa.

  • One can combine their own pre-processing code with ecoKI modelling BBs, for example, by defining a pipeline whose first step is an ecoKI data reader which loads the pre-processed data, then passes it to the ecoKI modelling BB by mapping the result of the data reader to the proper input of the modelling BB. If the pre-processing code is a customized building block, then one can define a pipeline that applies this building block, then the modelling BB sequentially.

4. How to use the visualization BB example to write the visualization method of individual BB? (essentially the answer will be instructions on how to use panel)

  • To implement a customized visualizer, one needs to define a new class that inherits from the ecoki Visualizer class. This class needs to define a run method, where data is fetched from the input dictionary, a Panel object – the interactive dashboard – is created, assigned to the self.visualizer attribute and the self._show_visualizer() method is called. The simplest Panel object that would be informative is to take a pandas DataFrame and call .interactive() on it, which returns an interactive Panel DataFrame (requires importing hvplot.pandas). This allows real-time interaction with the underlying DataFrame, which can be transformed, filtered, etc, while the created plot is updated automatically. Calling .hvplot() on this interactive Panel object creates a plot, which can be assigned to self.visualizer of the Visualizer sub-class created earlier. For any more detailed visualizations, one could take inspiration from the given visualizers or online Panel tutorials.Existing visualizers can also be used with customized BBs by setting the visualizer_module and visualizer_class BB entries to the visualizer’s module and class names when defining a pipeline in a settings.json file. The visualizer_input entry maps the BB output, i.e. the data to visualize, to the input of the visualizer (the dictionary containing the data to visualize). This is illustrated in the following example, where the local data reader BB uses the tabular data visualizer. "nodes": [ { "name": "local_data_reader", "building_block_module": "ecoki.building_blocks.code_based.data_integration.acquire_data.data_reader_local.data_reader_local", "building_block_class": "DataReaderLocal", "execution_mode": "local", "settings":{"data_file_path":"ecoki/datasets/tabular_data_nan.csv","index_name":"timestamp"}, "visualizer_module": "ecoki.building_blocks.code_based.data_integration.identify_and_understand_data.tabular_data_visualization.tabular_data_visualizer", "visualizer_class": "TabularDataVisualizer", "visualizer_input": {"input_data": "output_data"}, "interactive_configuration": true } ],

5. How to run the multiple methods (modules) in one BB

  • Ideally the purpose of a building block is to serve a singular purpose. If this functionality is complex and needs multiple function definitions, then these could be defined as class method inside the scope of class or as dynamic methods (monkey patching) out the class definition. .
DASHBOARD

1. How to initiate and run the dashboard application?

  • The ecoKI dashboard can be started by executing "ecoki\central_dashboard\app\start_dashboard.py". If no custom address or port was specified type “localhost:20000” in a browser. The ecoKI-Dashboard login-page should appear. You can sign in with the user "ecoki_test" and password "energy2022". For full functionality of the dashboard the backend and pipeline pool needs to be available. A full documentation for how to start all required services is provided in README.md..

2. How can we get parameters from user through the visualization class of dashboard for the specific building block?

  • To configure pipelines step by step and provide settings via the UI, a configuration gui mode for running the pipelines is available. Once you have created a pipeline under “Active pipelines”, select it and click on the “Configure run pipeline” button. You can now browse through the building blocks and configure them via the UI.

3. How to install ecoKI on Windows, Mac, Linux?

  • Go to the navigation bar item “Pipelines”. From the existing pipelines select the one you want to start and click on “View Details”. A page is opened where you get additional information. To create it click on “Create this pipeline” and it should appear in the navbar item “Active pipelines”.
DEPLOYMENT

1. How to install ecoKI on Windows, Mac, Linux?

  • a. Download ecoKI i. Install git, if it isn’t available on your system already. 1. See https://git-scm.com/downloads ii. Run in a command window / terminal to download ecoKI using git: 1. git clone https://gitlab.bik.biba.uni-bremen.de/hop/ecoki.git iii. (using the ZIP download on our Gitlab or a git GUI is possible too, but won’t be supported) b. Install Docker i. See https://docs.docker.com/engine/install/ c. Start ecoKI, it will install on first run i. Windows: Run “docker_start.bat” 1. You might have to start “Docker Desktop” beforehand to start the docker daemon ii. Linux / Mac: Run “docker_start.sh”

2. How to update ecoKI?

  • a. Open a command window / terminal inside the ecoKI folder b. Run the following command to update your copy: i. git pull c. Run ecoKI like described above d. In case of persisting issues / unexpected behavior: Force a rebuild of the environment i. Windows: Run “docker_start_clean.bat” ii. Linux / Mac: Run “docker_start_clean.sh”

3. Can ecoKI auto start?

  • a. Yes, to configure i. Oen the “compose.yaml” file in the ecoKI root folder ii. Change the “restart” setting, for example to “always” 1. To disable, set it to “no” 2. Available options: See https://docs.docker.com/config/containers/start-containers-automatically/#restart-policy-details iii. (Re)start ecoKI to apply
{% endblock %} {% block extra_foot %} {% endblock %} {% endif %}