pysep.opac.tops.api package¶

Submodules¶

pysep.opac.tops.api.api module¶

Author: Thomas M. Boudreaux

Created: September 2021

Last Modified: September 2021

Psuedo API for querying TOPS webform

pysep.opac.tops.api.api.TOPS_query(mixString: str, mixName: str, nAttempts: int) → bytes¶

Query TOPS form and retry n times

Parameters

mixString (string) – string in the form of: “massFrac0 Element0 massFrac1 Element1 …” which will be sumbittedi in the webform for mixture
mixName (string) – name to be used in the webform
nAttemptes (int) – How many times to reattempt after a failure.

Returns

tableHTML – Table quired from TOPS cite.

Return type

bytes

pysep.opac.tops.api.api.TOPS_query_async_distributor(aFiles, outputDirectory, njobs=10)¶

Distributes TOPS query jobs to different threads and gathers the results together. Writes out output.

Parameters

aFiles (list of TextIO) – List of file like objects to abundance files to be parsed
outputDirectory (str) – Path to directory to save TOPS query results to.
njobs (int, default=10) – Number of concurrent jobs to allow at a time.

pysep.opac.tops.api.api.call(aTable: str, outputDir: str, jobs: int)¶

Main TOPS psuedo API call function. Will save results to outputDir with file format OP:IDX_X_Y_Z.dat where IDX is the ID of the composition (parallel to DSEP composition ID), X is the classical Hydrogen mass fraction, Y is the classical Helium mass fraction, and Z is the classical metal mass fraction.

Parameters

aMap (str) – Path to a chemical abundance table with the form as described in the documentation for pysep.opac.tops.parseChemFile
aTable (str) – Path to chemical abundance table to be used as base composition.
outputDir (str) – Path to directory save TOPS query results into
jobs (int) – Number of threads to query TOPS webform on

pysep.opac.tops.api.api.parse_table(html: bytes) → str¶

Parse the bytes table returned from mechanize into a string

Parameters: html (bytes) – bytes table retuend from mechanize bowser at second TOPS submission form
Returns: table – parsed html soruce in the form of a string
Return type: string

pysep.opac.tops.api.api.query_and_parse(file: TextIO, outputDirectory: int, i: int, nAttempts: int = 10)¶

Async coroutine to query TOPS webform, parse the output, and write that to disk.

Parameters

file (TextIO) – Abundance file to be parsed for the form as defined in the docstring for the function parse_abundance_file
outputDirectory (str) – Path to write out results of TOPS webquery
i (int) – Index of composition so file name can properly keep track of where it is, even in parallel processing.
nAttempts (int, default=10) – Number of time to retry TOPS query before failing out

pysep.opac.tops.api.api.submit_TOPS_form(mixString: str, mixName: str, massFrac: bool = True) → bytes¶

Open the Los Alamos opacity website, submit a given composition and then return the resultant table.

Parameters

mixString (string) – string in the form of: “massFrac0 Element0 massFrac1 Element1 …” which will be sumbittedi in the webform for mixture
mixName (string) – name to be used in the webform
massFrac (bool, default=True) – Submit as massFrac instead of numberFrac

Returns

tableHTML – Table quired from TOPS cite.

Return type

bytes

pysep.opac.tops.api.convert module¶

Author: Thomas M. Boudreaux

Created: September 2021

Last Modified: September 2021

Main conversion code for TOPS api, responsible for takine many TOPS results and merging them into a single OPAL formate high temperature opacity file.

Functions¶

comp_list_2_dict
Take a list containing compsoition information for a star in the form of [(‘Element Symbol’, massFraction, numberFraction),…] and convert that into a dictionary of the form: {‘Element Symbol’: (massFraction, numberFraction),..}.

parse_RMO_TOPS_table_file
Given the path to a file queried from the TOPS webform put it into a computer usable form of 3 arrays. One array of mass density, one of LogT and one of log Rossland Mean Opacity

convert_rho_2_LogR
Maps a given kappa(rho,logT) parameter space onto a kappa(LogR, LogT) field through interpolation. The final field is the field that DSEP needs.

extract_composition_path
Given the name of a TOPS return file (named in the format OP:n_X_Y_Z.dat) extract X, Y, and Z

format_opal_comp_table
Take in all the information from a given TOPS tables and format it to the proper format for DSEP to undersand. Leave in some placeholders so that in future table can be labeld as the proper number.

format_OPAL_header
Writes the header of the opacity table that DSEP expects. This is written to be the same length (and basically the same contents) of the header from the OPACITY project. Not sure if that is required; however, if so I am matching it.

format_OPAL_table
Given a dictionary of tables and a composition Dictionary for solar composition in a given mixture (AGSS08, GS98, etc…) merge all the information together into a string which can be written to disk and would be the format of an opacoty project table (what DSEP expects)

format_TOPS_to_OPAL
Take the path to a table queried from the TOPS web form and fully convert it into a table which can be directly read by DSEP. (Note this function does not write anything to disk; however, the return products can be written to disk)

rebuild_formated_table
Iterate over a list of opacity tables and a list of desired chemical compositions then replace the contents of the table list with the newly updated RMOs frmo the interpolation.

TOPS_2_OPAL
Main conversoin utility to go between some set of TOPS tables and an OPAl table. Will take a set of 126 TOPS tables where each one is the opacity for one composition over a number of temperature and densities and rearange them into one large file with 126 tables within it. Each table will be over a range of temperatures and R values. To get to R val interpolation is used.

pysep.opac.tops.api.convert.TOPS_2_OPAL(outputDirectory: str, aTable: str, output: str)¶

Main conversoin utility to go between some set of TOPS tables and an OPAl table. Will take a set of 126 TOPS tables where each one is the opacity for one composition over a number of temperature and densities and rearange them into one large file with 126 tables within it. Each table will be over a range of temperatures and R values. To get to R val interpolation is used.

Parameters

outputDirectory (str) – Path to directory where TOPS query results are stored
aTable (str) – Path to a reference abundance table to use when filling the header with compositional information
output (str) – Path to save final OPAL formated table too

pysep.opac.tops.api.convert.comp_list_2_dict(compList: collections.abc.Iterator) → dict¶

Take a list containing compsoition information for a star in the form of [(‘Element Symbol’, massFraction, numberFraction),…] and convert that into a dictionary of the form: {‘Element Symbol’: (massFraction, numberFraction),..}.

Parameters: compList (Iterator) – list of the form: [(ElementSymbol, massFrac, numberFrac, ZmassFrac, massFrac Uncertanity, numberFrac Uncertanity, ZMassFrac Uncertanity),…]
Returns: Dictionary of the form {‘Element’:(massFrac, numberFrac, ZmassFrac, massFrac Uncertanity, numberFrac Uncertanity, ZMassFrac Uncertanity),…}
Return type: dict

pysep.opac.tops.api.convert.convert_rho_2_LogR(rho: numpy.ndarray, LogT: numpy.ndarray, RMO: numpy.ndarray) → Tuple[numpy.ndarray, numpy.ndarray, numpy.ndarray]¶

Maps a given kappa(rho,logT) parameter space onto a kappa(LogR, LogT) field through interpolation. The final field is the field that DSEP needs.

Parameters

rho (np.ndarray) – mass density array of size n
LogT (np.ndarray) – LogT array of size m
RMO (np.ndarray) – Opacity Array of size m x n

Returns

targetLogR (np.ndarray(shape=19)) – Log R values which dsep requires
targetLogR (np.ndarray(shape=70)) – Lof T values which dsep requires
Opacity (np.ndarray(shape=(70, 19))) – Opacity array now interpolated into LogR, LogT space from rho LogT space and sampled at the exact LogR and LogT values required.

pysep.opac.tops.api.convert.extract_composition_path(path: str) → Tuple[float, float, float]¶

Given the name of a TOPS return file (named in the format OP:n_X_Y_Z.dat) extract X, Y, and Z

Parameters

path (string) – path to TOPS return file

Returns

X (float) – Hydrogen mass fraction
Y (float) – Helium mass fraction
Z (Metal mass fraction)

pysep.opac.tops.api.convert.format_OPAL_header(compDict: dict) → str¶

Writes the header of the opacity table that DSEP expects. This is written to be the same length (and basically the same contents) of the header from the OPACITY project. Not sure if that is required; however, if so I am matching it.

Parameters: compDict (dict) – dictionary in the form: {‘Element’: (massFrac, numFrac), …} used to fill up the header with composition information. This is meant to be the “solar” compositon of whatever mix you are using so… [Fe/H] = 0.0, [alpha/H] = 0.0, a(He) = 10.93
Returns: The Header to be prepended to the opacity table file
Return type: string

pysep.opac.tops.api.convert.format_OPAL_table(tableDict: dict, compDict: dict) → str¶

Given a dictionary of tables and a composition Dictionary for solar composition in a given mixture (AGSS08, GS98, etc…) merge all the information together into a string which can be written to disk and would be the format of an opacoty project table (what DSEP expects)

Parameters

tableDict (dict) – dictionary of table elements, containing a “Summary” entry (metadata) and a “Table” entry. All occcurences of the the string “TNUM” will be replaced with the index+1 of where that table occurs in the file.
compDict (dict) – dictionary in the form: {‘Element’: (massFrac, numFrac), …} used to fill up the header with composition information. This is meant to be the “solar” compositon of whatever mix you are using so… [Fe/H] = 0.0, [alpha/H] = 0.0, a(He) = 10.93

Returns

OPALFormatted – Opacity Project formated table as a string which can be written to disk

Return type

string

pysep.opac.tops.api.convert.format_TOPS_to_OPAL(TOPSTable: str, comp: tuple, tnum: int, upperNonRect: Optional[numpy.ndarray] = None) → Tuple[str, float, float, float, numpy.ndarray, numpy.ndarray, numpy.ndarray]¶

Take the path to a table queried from the TOPS web form and fully convert it into a table which can be directly read by DSEP. (Note this function does not write anything to disk; however, the return products can be written to disk)

Parameters

TOPSTable (string) – path to table queried from TOPS web form
comp (dict) – composition dictionary
tnum (int) – table number
upperNonRect (np.ndarray, optional) – array describing how to fill the top of the table non rectangurally

Returns

metaLine (str) – metadata extracted from table
X (float) – Hydrogen mass fraction
Y (float) – Helium mass fraction
Z (float) – Metal mass fraction
LogR (np.ndarray(shape=19)) – Log R values which dsep expects
LogT (np.ndarray(shape=70)) – Log Temperature values which dsep expects.
LogRMO (np.ndarray(shape=(70, 19))) – Log rossland mean opacities for the LogT and LogR arrays

pysep.opac.tops.api.convert.format_opal_comp_table(LogR: numpy.ndarray, LogT: numpy.ndarray, LogRMO: numpy.ndarray, TNUM: int, comp: Optional[dict] = None, upperNonRect: Optional[numpy.ndarray] = None) → Tuple[str, str]¶

Take in all the information from a given TOPS tables and format it to the proper format for DSEP to undersand. Leave in some placeholders so that in future table can be labeld as the proper number.

Parameters

LogR (ndarray) – The Log R value array (horizontal axis of table)
LogT (ndarray) – The Log Temperature value array (vertical axis)
LogRMO (ndarray) – all of the RMO values associated with R and T
TNUM (int) – Table number
comp (dict, optional) – composition dictionary. If not provided placeholders are left in place so that it may be filled later on
upperNonRect (ndarray, optional) – array describing how to fill the top of the table non rectangurally

Returns

metaLine (str) – header line for each table, may or may not have placeholders in it
fullTable (str) – full table to be places in opacity file

pysep.opac.tops.api.convert.parse_RMO_TOPS_table_file(TOPSTable: str, n: int = 100) → Tuple[numpy.ndarray, numpy.ndarray, numpy.ndarray]¶

Given the path to a file queried from the TOPS webform put it into a computer usable form of 3 arrays. One array of mass density, one of LogT and one of log Rossland Mean Opacity

Parameters

TOPSTable (string) – Path to file queried from TOPS webform
n (int, default=100) – The size of density grid used in TOPS query form.

Returns

rho (np.ndarray(shape=n)) – Array of mass densities (in cgs) parsed from TOPS table.
LogT (np.ndarray(shape=m)) – Array of temperatures (in Kelvin) parsed from TOPS table.
OPALTableInit (np.ndarray(shape=(m,n))) – Array of Rossland Mean Opacities parsed from TOPS table.

pysep.opac.tops.api.convert.rebuild_formated_tables(formatedTables, interpRMO, pContents, upperNonRect)¶

Iterate over a list of opacity tables and a list of desired chemical compositions then replace the contents of the table list with the newly updated RMOs frmo the interpolation.

Parameters

formatedTables (list of dicts) – List of dictionaries holding three axis each. X, Z, and LogRMO. These are the “observed” values to be interpolared
interpRMO (list of ndarrays) – RMOs after interpolation to LogR-LogT space from rho-LogT space.
pContents (np.array(shape=(n, 3))) – Numpy array of all the compositions of length n. For a dsep n=126. Along the second axis the first column is X, the second is Y, and the third is Z.
upperNonRect (ndarray) – Array describing how to fill the top of the table non rectangurally

Returns

formatedTables – List of dictionaries holding three axis each. X, Z, and LogRMO. These have been updated to reflect the compositions in pContents.

Return type

list of dicts

pysep.opac.tops.api.utils module¶

Author: Thomas M. Boudreaux

Created: September 2021

Last Modified: September 2021

Utilities to help with the TOPS query api

pysep.opac.tops.api.utils.format_TOPS_string(compList: list) → str¶

Format the composition list from pasrse_abundance_file into a string in the form that the TOPS web form expects for a mass fraction input.

Parameters: compList (list) – composition list in the form of: [(‘Element’, massFrac, numFrac),…]
Returns: TOPS_abundance_string – string in the form of: “massFrac0 Element0 massFrac1 Element1 …”
Return type: string

pysep.opac.tops.api.utils.validate_extant_tables(path: str, prefix: str) → bool¶

Check if there is a quiried table from TOPS for every number frac file generated by the program passed to call_num_frac.

Parameters

path (string) – Path to where the results of the number frac and TOPS query files are stored
prefix (string) – start prefix given to all abundance / number frac files

Returns

validated – Whether or not all number frac files have a corresponding TOPS opacity table

Return type

bool

Module contents¶

Author: Thomas M. Boudreaux

Created: September 2021

Last Modified: September 2021

The TOPS webform at https://aphysics2.lanl.gov/apps/ does not expose any public API. However, their website is amenable to scraping. This module implments a web scraper with the goal of automatically generating new high temperature opacity files.

Because the TOPS form uses Javascript a simple HTTP request cannot be built. Rather, the module mechanize is used to evaluate returned javascript which can then be used to call the acual returned HTML table.

There are two main sections to this api

The query
The conversion

The query section does the actual querying. The conversion section should arguably be its own thing; however, its here for now. The conversion section converts many TOPS tables into a single OPAL (format DSEP needs) high temp opacity table.

Here are a few usage examples, for the following examples there is some file called abun in the current working directory which is in the form of a chemical abundance file as defined in the documentation for parseChemFile. Finally, there is also some directory called ‘out’ in the current working directory’ which we will be saving results to.

Calling the TOPS webform. The first thing you need to do is to generate a function which maps X, Y, and Z to Fe/H and a(He). You can do this yourself, or you can use the built in function which should be good for most usecases.

>>> from pysep.misc.abun.utils import gen_abun_map
>>> aMap = gen_abun_map('abun')

Now you have an abundance map, we can now call the query code

>>> from pysep.opac.tops.api import call
>>> call('abun', 'out', 25)

Where 25 is the maximum number of concurrent connections we want to allow the scrapper to make to the TOPS website. If this is too low the code can take nearly 2 hours to generate a single table. However, be careful not to make it too high or the webserver could kick you for malicious use. I have found that 20-30 concurrent connections seems to be good, and lets a single table generate in between 5-8 minutes.

However, after the last section ran we dont yet have a table. For that we need to invoke the conversion utility code

>>> from pysep.opac.tops.api import TOPS_2_OPAL
>>> TOPS_2_OPAL('out', 'abun', 'finalResults.txt')

This will leave a file called ‘finalResults.txt’ which is the OPAl formated high temperature opacity file queried from TOPS