Initial DataSet Creation

Time to flood the system with DataSets

There exists a situation where we have to execute code before I can really dive into a concept. Fortunately, these happen rarely. Unfortunately, this is one of those times.

Briefly, a Subject is a higher order of data we will be modeling. A Subject consists of at least 5 tables, 20 views, and functions / procedures. We have talked about a few subjects so far: Databases, DataSets, DataPoints. This concept allows us to tell the system to create a Subject that we need to model with a single classified record, and the system creates all of our objects.

I will be spinning up 38 subjects, for a total of 460 tables. Only a few matter at this very moment. I have created a .json file that I can use in python to spin up this system using the /code/dataset idea.

The Subjects that matter right now are: Database, DataSet, DataPoint, and Group. This is because we need to get our own information_schema populated so we can automate system tasks.

Code

I have my own .json file to spin up systems very quickly. In the beginning (right now), this is what I do. I flood the system with objects, run a few functions to get their defaults and states in place, and am ready to start building what the business needs.

We could abstract these statements with additional logic, but (dev) speed is important.

1
2
3def create_table(_conn,_obj,_scripts):
4   _name = _obj['name']
5   _namespace = _obj['namespace']
6   _extras = _obj['extraCols']
7   _basetype = _obj['basetype']
8   _extra_col =''
9   _extra_cons = ''
10   for i,x in enumerate(_extras):
11      if x['location']=='column':
12         _extra_col = _extra_col + ','+x['value']
13      if x['location']=='constraint':
14         _extra_cons = _extra_cons + ','+x['value']
15      
16   create_base = _scripts['DATASET BASE']
17   create_base_rel = _scripts['DATASET BASE RELATIONSHIP']
18   create_type = _scripts['DATASET TYPE']
19   create_family = _scripts['DATASET FAMILY']
20   create_class = _scripts['DATASET CLASS']
21   create_realm = _scripts['DATASET REALM']
22
23   ex_base = create_base.replace('<<EXTRA COLUMNS>>',_extra_col).replace('<<EXTRA CONSTRAINTS>>',_extra_cons).replace('<<NAMESPACE>>',_namespace).replace('<<NAME>>', _name)
24   ex_base_rel = create_base_rel.replace('<<EXTRA COLUMNS>>',_extra_col).replace('<<EXTRA CONSTRAINTS>>',_extra_cons).replace('<<NAMESPACE>>',_namespace).replace('<<NAME>>', _name)
25   ex_type = create_type.replace('<<NAMESPACE>>',_namespace).replace('<<NAME>>', _name)
26   ex_family = create_family.replace('<<NAMESPACE>>',_namespace).replace('<<NAME>>', _name)
27   ex_class = create_class.replace('<<NAMESPACE>>',_namespace).replace('<<NAME>>', _name)
28   ex_realm = create_realm.replace('<<NAMESPACE>>',_namespace).replace('<<NAME>>', _name)
29   
30   cur = _conn.cursor()
31   cur.execute(ex_realm)
32   cur.execute(ex_class)
33   cur.execute(ex_family)
34   cur.execute(ex_type)
35   if _basetype!='ATTRIBUTE':
36      if (_basetype=='STANDARD' or _basetype=='DOMAIN'):
37         cur.execute(ex_base)
38      if _basetype=='RELATIONSHIP':
39         cur.execute(ex_base_rel)
40   _conn.commit()
41
42

There are 2 .json files involved: One contains records for a parameterized create table, the other.. those 38 subjects.

If you are looking for these scripts and files / notebooks.. find me on linked in and send me a message. A lot of thought has went into the subjects and what they represent, a lot of thought.