Based on my previous article on running Python/R on Android, it must be obvious that I started using my Android based tablet for development. Things were going well till I decided to start training/evaluating LLMs locally. The last laptop I purchased in late 2018 was a Lenovo Legion Y530 (or maybe Y730). It a dedicated NVIDIA GeForce GTX 1050-Ti GPU (I used it primarily for training deep learning models; I’m past my gaming days) and cost around USD 1,000 - it was the best price-to-performance laptop for my planned budget. This time my budget was flexible, but the constraint depended on the VRAM needs of LLMs. Unfortunately, the more budget friendly (compared to NVIDIA GPUs) AMD Radeon GPUs aren’t the target of most ML/DL frameworks.

After doing some research I narrowed my choices to laptops that have a dedicated RTX 4080 or RTX 4090 GPU. I thought I nailed my laptop needs and decided to be price conscious - to minimize price-to-performance if possible. So I shortlisted a few models, studied their reviews in detail and started looking . But I never expected the wild goose chase that followed (this is coming from someone who followed people who provided stock updates and got on Target/Walmart/Amazon/GameStop/BestBuy websites at 7AM to buy Xbox Series X / Playstation 5).

Note: Unless specified, all prices include Massachusetts sales tax. Price may vary in your state depending on taxes.

Priorities

I narrowed down to the following configuration:

• NVIDIA RTX 4090 GPU with 16 GB VRAM (>130W) or NVIDIA RTX 4080 GPU with 16 GB VRAM (>115W)
• 32+ GB DDR5 memory
• Intel Core i9-13900HX+ for a ‘regular’ laptop or Intel Core Ultra 7-155H+ for lightweight, power conserving laptop
• Display size: 16” or lesser (only to minimize weight)
• Should support Linux (apparently some laptops don’t, I don’t know how they do it) + Windows dual boot
• Decent heat management
• Ideally light weight (preferably less than 5 lbs without the charger)

Clearly, it’s impossible to find a laptop that satisfies all the criteria. My priorities in the order of importance: #5, #1, #2, #7, #6, #3, #4

Entering the Market

I initially planned to buy a laptop with RTX 4070 (8 GB VRAM in laptops), but quickly dropped the idea because:

• Many LLMs won’t work locally
• There was a significant jump in performance from RTX 4070 to RTX 4080

After applying some these filters on Best Buy it was clear that the prices were typically USD 2,000+ for RTX 4080 (finding something below USD 1,700 is very rare) and USD 2,500+ for RTX 4090 laptops (finding something below USD 1,900 is very rare). With the entry to RTX 50xx in late February these prices were expected to fall, but they did not (they still haven’t dropped). Going from USD 1,000 to USD 2,500 in a span of 7 years was shocking to me. So, I decided to look for alternatives on Google Shopping. After spending a few days on filtering (it’s challenging because Google Shopping feels almost exclusively LLM driven), I narrowed down to a few laptop models.

Problems

Buying a laptops with a dedicated GPU at a fair price isn’t easy:

• They typically don’t have a standard price - price changes from website to website despite a ‘price match guarantee’
• Even if a batch of laptops get released, they usually sell out very quickly (not as quickly as Xbox Series X / Playstation 5, but one should keep the prices in mind for this comparison - USD 2,000+ vs. USD 500)
• Some legal retailers allow ordering even if the laptop is on backorder (Eg: Adorama, CDW, Direct Dial, etc.). They’re usually priced lower than Amazon or Best Buy (where you can order instantly), but the price may change before the order ships (they mention how long the price is valid, but usually the laptops won’t ship by that date because of supply constraints)

Personally, I’ve been an ‘early visitor’ when a new batch was release - not by chance, but because I visit those pages frequently. But I failed to pull the trigger because of the price (Eg: USD 1,750+ for RTX 4080 laptops). Simultaneously, I placed an order on a backordered RTX 4080 laptop for ~USD 1,500 - this order never materialized. Finally, I decided to opt for used laptops as well. Enter Facebook Marketplace and eBay. Maintaining all the informatoin in one place was becoming difficult with increase in number of laptop models, configurations, prices in websites and conditions.

Solution

I decided to build a structured database to store all the information described above. In addition I decided to the structured database I decided to build an app to enter data manually and a dashboard with plots and filters to meaningfully visualize the data. This gave me an idea about the lowest price I can pay to get a laptop with RTX 4080 or RTX 4090 GPU.

Link to dashboard repo: https://github.com/SNaveenMathew/laptop_price_tracker

There isn’t much to say about the design of the app itself - the whole data entry process was manual, but I only had to add the ‘changes’. So, once all the models and sources were listed, I only had to enter data if the price changed. Even with this basic database design (including the uncertainty of transient eBay auction prices) I was able to obtain better price insights than Google Shopping (of course, I was using all the relevant sources from Google Shopping and beyond)!

Learnings

One will inevitably arrive at a few of the following conclusions:

• Budget laptops don’t exist at the higher end of performance
• Laptops at the higher end of performance don’t sell at a fair price
• It’s possible to buy a high performance laptop at a price that’s not abnormally high compared to its fair price
• Performance and weight are opposing forces - heating is the fundamental cause of this issue
• A laptop on backorder that can be purchased (first-come-first-serve) should only be a backup; it should not be a primary choice
• Price war on eBay auctions starts on the last day

I was alternating between #2, #3 and #5 for a long time. I kept the orders on backordered laptops open as long as the price was fixed even though the orders never shipped - this was just a backup. Since I had the sources, I found a ‘Used - Like New’ RTX 4090 laptop that was cross listed on Facebook Marketplace (USD 2,025) and eBay auctions (priced USD 1,650 with 2 days left). The seller accepted an offer of ~USD 1,800.

There were some laptops that I missed at the cost of collecting data. For example, one ‘Used - Like New’ RTX 4080 laptop sold for ~USD 1,400. I don’t regret missing those; it is the opportunity cost of obtaining accurate price insights in a corrupt market full of scalpers.

The laptop was in very good shape, and I received some goodies such as a cooling pad a liquid coolant. As of 3/20/2025 I did not find a better price for a laptop with RTX 4090 GPU.

PS: I’m not an expert, but I’m happy to name a few laptop models that amazed me.

Closing notes:

1. I’ve always been in favor of desktops over laptops; the only 2 reasons for choosing a laptop over a desktop are: 1. laptops can be used anywhere; desktops should remain on the desk, 2. laptops can be light enough to carry (most aren’t; I’m not including Macs in the discussion here as price-to-performance isn’t good)
2. If performance is the only factor, and if one doesn’t mind paying extra for electricity, one should consider desktops over laptops

Disclaimer: If one has the option, one should use rstudio.cloud and shinyapps.io to develop and deploy RShiny apps. Following this article is a waste of time.

Working on R/Python in Android OS may sound like an artificial constraint, but I faced this scenario - I wanted to run a RShiny app, but I did not have the resources to run apps in the ‘free tier’ both locally and on the Cloud.

Experts claim Android OS is just a version of Linux, but I’m not an expert in this area. This short article covers how a layman (like me) can run R scripts in Android OS. Similar steps can be followed to install and run Python in Android OS. Clearly, this is not an everyday scenario; not even something worth trying for fun.

Install Termux

DO NOT INSTALL FROM GOOGLE PLAY STORE - the version on Google Play Store was not updated in years, and lacks several basic Linux functionalities like apt install

1. Download and install F-Droid - it’s free; the APK is available here
2. Install Termux using F-Droid

Failed Effort - using its-pointless to install R

Note: Steps are documented in this Reddit comment. Some people claimed to be successful. So I’ll add a disclaimer - “Your results may vary”, but I hit a dead-end after crossing several roadblocks. The high-level steps include

1. Add the its-pointless repo
2. Install the necessary tools/software like gcc, gfortran, etc.
3. Download the source code for R
4. Use make install to build and install R

Successful - using Termux-Ubuntu

1.Follow the instructions in [this README] (https://github.com/Neo-Oli/termux-ubuntu/blob/master/README.md)

1. Run ./start-ubuntu. You’re now a root user
2. Follow the normal steps used to install R on Ubuntu (similar steps for Python)

I’m not sure if F-Droid can be uninstalled after this step. I will update this article once I’m in a position to experiment.

Final Step - running RShiny apps

It’s possible to setup Ubuntu like windowing system for Ubuntu in Android OS. Once again, some people claimed to be successful using the steps in this Reddit post. I tried this, and also tried installing x11 using apt install, but neither solution worked for me. So, I’ll add another disclaimer - “Your results may vary”, but I hit a dead-end.

Instead of setting up a windowing system like x11, I ran the RShiny app using Rscript command - this doesn’t pop-up a new window with the app, but the app can be opened in any browser by entering “localhost:". To avoid randomness in the port number it's better to run `Rscript -e "shiny::runApp(host='127.0.0.1', port=)"`.

Now we have R (or Python) running in Android OS. Tablet hardware (especially CPUs) are meant to conserve energy, so they may be extremely slow with compilation and program execution (as you might’ve noticed if you followed along with the installation of tools/software like gcc in Termux-Ubuntu). It’s time to tackle some interesting data-driven problems (I’m pursuing one - again, it’s not for everyone; more on this soon).

I found this meme on LinkedIn recently. After reading this post I hope people feel the same about python.

Experimenting with ‘AND’

Let’s assume we have a simple function in python to check if all the numbers in a list are greater than a threshold. Logically the function should return False if any number in the list violates the condition. Therefore, a single False is sufficient to make all([…, False, …]) return a False. Similarly, a single True is sufficient to make any([…, True, …]) return a True. Therefore, if ‘n’ is the length of the list, the number of steps required to determine the output should be strictly less than or equal to n. However, things may become interesting if we use a recursive function instead of a loop to obtain the result. Consider the following function:

def all_greater(nums, threshold):
  len_nums = len(nums)
  print(1)
  if len_nums == 1:
    return nums[0] > threshold
  else:
    return (nums[0] > threshold and all_greater(nums[1:], threshold))

all_greater([1], 0)

Output:

1
True

It is well known that recursive functions use a “call stack”. Consider the following function call and try to reason how many times print(1) will be executed:

Surprisingly (or unsurprisingly), the answer is — just once! Let’s change the order of conditions under ‘else’ and see what happens:

def all_greater(nums, threshold, num_calls = 1):
  len_nums = len(nums)
  if num_calls>=100:
    print(num_calls)
  if len_nums == 1:
    return nums[0] > threshold
  else:
    return (nums[0] > threshold and all_greater(nums[1:], threshold, num_calls = num_calls + 1))

all_greater([x for x in range(100)], 100)

Output:

False

def all_greater(nums, threshold, num_calls = 1):
  len_nums = len(nums)
  if num_calls >= 100:
    print(num_calls)
  if len_nums == 1:
    return nums[0] > threshold
  else:
    return (all_greater(nums[1:], threshold, num_calls = num_calls + 1) and nums[0] > threshold)

all_greater([x for x in range(100)], 100)

Output:

100
False

Surprisingly, the function was called 100 times! Despite the fact that and([A, B]) == and([B, A]), and nums[0] > threshold is violated for the very first index, the recursive call was made to the function. Therefore, we can conclude that the statement on the left is given preference during the evaluation of ‘and’ condition.

Experimenting with ‘OR’

def any_greater(nums, threshold, num_calls = 1):
  len_nums = len(nums)
  if num_calls >= 100:
    print(num_calls)
  if len_nums == 1:
    return nums[0] > threshold
  else:
    return (all_greater(nums[1:], threshold, num_calls = num_calls + 1) and nums[0] > threshold)

all_greater([x for x in range(100)], 100)

Output:

100
False

Unsurprisingly, the same pattern is observed with True and ‘OR’. However, it only gets more mysterious from here.

Enter numpy

More experiments with ‘AND’

import numpy as np
False and np.nan
np.nan and False
True and np.nan
np.nan and True

Outputs:

False
False
nan
True

This arises naturally from the following:

False and A = A and False = False

True and A = A and True = A

More experiments with ‘OR’

import numpy as np
False or np.nan
np.nan or False
True or np.nan
np.nan or True

Outputs:

nan
nan
True
nan

False or A = A or False = A

True or A = A or True = True

Surprisingly, np.nan or True -> nan! This contradicts A or True = True. At present I’m unable to think of a reasonable explanation for this result.

The mystery does not end there

import numpy as np
np.logical_and([False]*10, [np.nan]*10)
np.logical_and([np.nan]*10, [False]*10)
np.logical_and([True]*10, [np.nan]*10)
np.logical_and([np.nan]*10, [True]*10)
np.logical_or([False]*10, [np.nan]*10)
np.logical_or([np.nan]*10, [False]*10)
np.logical_or([True]*10, [np.nan]*10)
np.logical_or([np.nan]*10, [True]*10)

Outputs:

array([False, False, False, False, False, False, False, False, False, False])
array([False, False, False, False, False, False, False, False, False, False])
array([True, True, True, True, True, True, True, True, True, True])
array([True, True, True, True, True, True, True, True, True, True])
array([True, True, True, True, True, True, True, True, True, True])
array([True, True, True, True, True, True, True, True, True, True])
array([True, True, True, True, True, True, True, True, True, True])
array([True, True, True, True, True, True, True, True, True, True])

Mysteriously, all the statements resulted in True / False outputs. The most intriguing results were:

import numpy as np
np.logical_or(False, np.nan)
np.logical_or(np.nan, False)

Outputs:

True
True

While np.nan.bool() -> True explains all these observations, it does not explain why np.nan or True -> nan instead of True (simplified).

Conclusion

Logical ‘OR’ and ‘AND’ statement in python seem to execute the statement on the left before executing the statement on the right. “A and False = False and A = False” and “True or B = B or True = True” can reduce the number of call stacks in a recursive function if the output is simply ‘and([List])’ or ‘or([List])’.

Handling missing boolean values in numpy is not very intuitive. Using numpy’s logical_and / logical_or may lead to inconsistent outputs that may pass unit tests, but can lead to incorrect results during runtime.

Additional Reading

Please look at the following articles for more interesting observations on python’s handling of logical expressions:

• https://peterbbryan.medium.com/3-curiosities-of-python-booleans-c45379827c6c
• https://dbader.org/blog/difference-between-is-and-equals-in-python

Kepler telescope was launched in the year 2009 into an Earth-trailing heliocentric orbit. The planned lifetime of the mission was 3.5 years, but the mission continued for almost 10 years. The objective was to scan a portion of the sky to identify Earth-like habitable exoplanets in a region of the sky within the Milky Way galaxy. During its lifetime Kepler scanned the light curves of over 530,000 stars and detected 2,360 exoplanets (updated on 06/16/2020). Finally, Kepler mission was retired in 2018.

TESS and beyond

Transiting Exoplanet Survey Satellite (TESS) was launched in 2018. It was designed to search for exoplanets using the transit method in an area 400 times larger than that covered by the Kepler mission. TESS will provide candidates for further characterization by the James Webb Space Telescope and other telescopes. Unlike previous surveys which detected giant exoplanets, TESS is expected to find a large number of small planets around the nearest stars in the sky.

The following missions have been proposed to detect exoplanets (not all of these will use the transit method):

• CHEOPS
• JWST (known to many by its abbreviation — James Webb Space Telescope)
• PLATO
• ARIEL
• WFIRST

Heterogeneity in data sources

Sampling rates of different telescopes may be different. Kepler light curve data set has a sampling time period of ~ 20.4 ms. TESS telescope has a sampling time period of 2 seconds. This difference in sampling time periods leads to difference in amount of data history required for inspecting the transit light curve. Therefore, a supervised machine learning model trained on Kepler data cannot be applied to TESS light curve.

Transit method and ‘automation’ using splines (statistics)

Transit method is commonly used in astronomy for exoplanet candidate identification. The following image shows the essence of transit method:

A cup-like dip in the light curve shows the presence of an exoplanet around a star. This is the theoretical version; an ‘ideal’ practical version after data augmentation can be found below:

The data is very noisy even after data augmentation. For people who are familiar with statistical modeling — fitting a histogram regression or any other type of spline on this data requires careful choice of knots, which requires manual supervision — it is not fully automatic. Using automatic knot selection methods such as smoothing spline (+ hyperparameter tuning, cross-validation) may lead to a flattened out estimate even on the augmented light curve. This creates a problem — many candidates will be ignored.

Other end of the spectrum — manual tagging

The above animation shows a relatively easy example. There are 2 operations:

1. Multiple brushes on the original image to designate a region
2. Zooming in and refining each region

The manual complexity of this task is O(c * n), where n is the number of transits observed and c is the average number of manual operations (such as brushes, clicks, etc.) per transit. Usually c > n for most light curves. Even for a simple task with n = 3 (as shown above), this takes several minutes per light curve.

Full scale of the problem

There are billions of stars in the Milky Way galaxy. There are billions of galaxies like the Milky Way, each having billions of stars on an average. However, there are only a few thousand people who support tasks on Zooniverse. Therefore, the project is run in phases; few candidates from each phase are given to other projects for confirmation. NASA states that until now TESS has identified 1913 candidates out of which 51 have been confirmed (updated on 06/16/2020). This is a remarkable achievement in itself.

Can this task be simplified?

1. Brushing, zooming and refining are time consuming tasks. They are needed because the current set of methods are not robust to ‘noisy tagging’ — for example: terminating the task after brushing (without zooming and refining) leads to light curve of candidate + default light curve of the host star.
2. Can this be simplified further? Sections of the image can be automatically brushing (using unsupervised learning) and labeled as candidates. This process can be repeated for all light curves. It is alright to have few false positives, but the algorithm should produce much fewer false negatives. Manual taggers only have to click on the ‘x’ button on each brush or leave the light curve unaltered (approval that the automatic candidate identification is accurate). Brushing, zooming and refining will still be available to taggers, but the hope here is to reduce their usage.

• If a light curve has ‘large’ number of taggers, their approval and disapproval (+ optional refining) can be used to denoise the zone in which the candidate is present
• If the light curve has ‘small’ number of taggers, the default set of candidates tagged by the algorithm may be considered

Therefore, unsupervised learning can expedite the process of candidate identification. Eventually each ‘zone’ of the light curve and each user who tags will be given a credibility score. This score varies with each newly available example of ‘modified’ manual tagging. Eventually, the most credible candidates can be examined further or can be used to build a supervised learning model.

Conclusion

I strongly recommend people to support Planet Hunter TESS and other Zooniverse projects. Many projects require support from a large pool of enthusiastic individuals (like the readers who have reached this section). Crowdsourcing in scientific research will grow exponentially and I recommend the readers to be one of the early members of this growing community.

The previous section of article presented the vision for my independent research — unsupervised learning and crowdsourcing for exoplanet candidate identification. I’m currently developing the idea on my own, so progress has been slow. I welcome people to join me.

Note: This project will always remain open source.

Riding on the machine learning wave and the associated business jargon is an easy task. Nowadays the process of ‘applied data science’ has been limited to importing packages, reading data, preprocessing and using predefined APIs for classification/regression. The objective of this article is to use only the basic terminology associated with reinforcement learning, thereby limiting the target audience to people who are familiar with the underlying difficulties of defining a reinforcement learning problem. Expect a dry article because the details of the problem have been abstracted out.

Background — Personal

Summer internship can be tiring, but I feel the opposite. In the end, the ‘result’ is a binary flag which currently has a value False. This is one of the short run motivations. The long run motivation is philosophical. I rejected 2 (higher paying) offers to signup for a dream — to accomplish something significant in life. Dreams don’t come true in 3 months, but there were significant learnings during Summer.

Initial Work

The problem (confidential) was framed appropriately to fit into a reinforcement learning framework. The initial experiments were designed using DQN to approximate the Q function. These were immediately ruled out due to the large run time (estimate: ~ 20 days per experiment). Cleverly designed supervised bootstraps were used to instantiate the DQN, but these efforts failed because of the following reasons:

1. Sparsity of actions
2. Low signal-to-noise ratio

Through some exploration it became increasingly clear that the following fixes cannot solve the problem:

1. Changing the architecture of the body of DQN
2. Using duck tape solutions such as bootstrapping

Data Discovery

Reinforcement learning agent is designed to learn a policy that maximizes the expected reward. If the problem is well defined the feasibility of learning depends on the structure of the data and the effectiveness of features / feature engineering. During data exploration it was observed that there were special points of interest during an episode and that the locus of these points across different agents formed an interesting structure. The overall idea can be summarized as:

1. These interesting structures can be used to guide the agent. A small reward can be given once the agent reaches these structures. This also helps in the explainability of the reinforcement learning model.
2. Tweaks to the reward and differential rewards can be used to switch between a guided agent and an exploratory agent.
3. Approximate Q function can be used to avoid path-following behavior. High penalty can be used to apply physical reachability constraints.

Simpler Solution

If the transition probabilities are known, a model free approximation of the Q function can be computed based on historical data. The agent can be made to act in a greedy way (immediate reward, without exploration) to achieve its goal. However, it should be noted that a set of locally optimal paths is not guaranteed to build a globally optimal path. Therefore, this method is not guaranteed to work in environments that are drastically different from those observed in the past.

Simpler Solution

If the transition probabilities are known, a model free approximation of the Q function can be computed based on historical data. The agent can be made to act in a greedy way (immediate reward, without exploration) to achieve its goal. However, it should be noted that a set of locally optimal paths is not guaranteed to build a globally optimal path. Therefore, this method is not guaranteed to work in environments that are drastically different from those observed in the past.

Looking Forward

Knowing the transition matrix of historically successful episodes can simplify the state space drastically. Instead of acting greedily on immediate rewards at each state the agent can be made to act greedily on expected rewards (computed from historical data) at each state. [I have many unrefined ideas in mind, which are beyond the scope of this article]

Closing Remarks

Transition matrix incorporate knowledge about successful cases from the data. Unfortunately, reinforcement learning is not mature enough to learn high level concepts (such as the ‘locus of points of interest’) from the data. As a result, applying reinforcement learning to real life problems is extremely difficult.

I once again stress on the fact that it is easy to ride on the ML wave; it is extremely difficult to create one. Reinforcement learning community is perfecting Atari games, but lacks test beds for real problems. Once the first real problem is solved using reinforcement learning, this field is expected to dominate the near future. I encourage everyone to be a part of the wave; be the makers; don’t just ride on the wave.

Update: Oct 17 2024

I recently extended the analysis, but there are several more ideas for ‘discretization’.

Exoplanets are planets that are outside the Solar System. There are several types of exoplanets — classified by size and composition as Earth-size, Earth-like, Super-Jupiters, gas giants, rocky worlds the size of Earth, rocky giants, Super-Earths, mini-Neptunes, and gas dwarfs. Also, planets may revolve around a host star (like Earth around the Sun) and may be rogue planets that don’t have a companion star. Machine learning can be used in conjunction with astrophysics to automate the process of identifying exoplanet candidates. It is extremely difficult to automate the detection of rogue planets because they are very far away from stars and their motion around a given star is not periodic. It is possible to automate the identification of exoplanets that revolve around a host star.

Astrophysics behind Exoplanet Detection

Light curve tracks the intensity of light of a celestial object vs time. The first candidate for an exoplanet, Gamma Cephei Ab, was proposed in 1988. The first confirmed exoplanet was PSR B1257+12, a planet with approximately the same mass of Earth, orbiting a millisecond pulsar PSR B1257+12. Several exoplanets have been detected thereafter.

Kepler space telescope (2009–18) was launched to discover planets that are orbiting other stars. Transit photometry (shown above) is used to identify exoplanet candidates. But these detection using transit photometry involves several man-hours of manual supervision by astronomers and enthusiasts.

Why Unsupervised?

Kepler’s light curve data is publicly available. Machine learning models can be built to identify exoplanet candidates. Several attempts have been made to use supervised machine learning to the problem of exoplanet detection using several thousand tagged examples. However, tagged examples are limited by the number of man-hours for supervision. It is better to use man-hours in a wise way: by filtering as much as the data as possible and providing candidates that can be investigated further. This can be accomplished by unsupervised learning, which does not introduce human biases that are involved in supervision (tagging).

Machine Learning Problem

Definition

The objective is to identify patterns that correspond to drop in intensity of light for a significant amount of time. As an example, if Solar System were to be observed from a different star system, the total transit time of the Earth around the Sun is no more than 0.54 days; only the Sun’s light along with it’s trends and periodic changes (as seen from star system) will be observed during the remaining 364.7 days. Therefore, exoplanet candidate identification can be thought of as identification of periodic anomalies in the light curve.

Issues

The magnitude of drop in intensity may not be the same for all transits because the orbit of the planet as viewed from Earth is likely to precess. In other words, the azimuth of transit of the planet with respect to the start (as viewed from Earth) is likely to change between transits.

External Motivation

LSTM autoencoders have been used in time series anomaly detection IoT applications. After training, the autoencoder is able to encode-decode the regular signal almost perfectly, but fails to encode-decode the irregular (anomalous) signals with good accuracy. After removing the faults, the model can be incrementally retrained with the data from remaining components. This idea has been applied in industrial IoT with relatively good success. Since exoplanet detection also involves detection of rare, anomalous behavior in light curve, LSTM autoencoder can be used to solve the problem.

Initial Results

I executed this idea during Spring 2019 semester at UIUC. I could apply it to only 30 light curves because of resource constraints (CPU only training). Initial results suggest that candidate identification is possible without much scope for false negatives. However, detection without astronomy context led to several false positives. This was reduced using phase detection (folding). It was observed that the astronomy context (periodicity of planet transits) provided by phase detection was important for reducing false positives.

Final Words

This is definitely not the end of the road for this project. I have discussed my ideas for extending the project in this GitHub page.

Astronomy and physics have always fascinated me because the models in these fields go beyond machine learning and deal with fundamental problems such as universality, causality and forecast; all within the framework of the scientific method: hypothesize, test, argue why models that fit the data actually work, discard ideas that don’t fit the data, repeat. I hope we eventually identify ways to tackle the problem of reasoning in machine learning. If this happens in the future, machine learning and natural sciences will work together (the movement has already started).

I’m currently pursuing independent research on large financial data. Data engineering is an integral part of the analysis as financial data is often not in the format required for analysis. Financial data may be provided in the form of transactions/updates or in the form of aggregates. While it is easy to go from transaction level to aggregate level, it is difficult to do the opposite.

My laptop configuration is decent: Ubuntu 18.04, 16 GB DDR4, Intel core i7–8750H (6 + 6 virtual core) @ 2.2 GHz and a GPU (not relevant here). But brute force is not a good idea — code run time was ~ 4 hours for processing one day’s data, which is not acceptable.

Data (samples):

Without mentioning the type of data, here are few anonymized samples (dy’s can be positive or negative):

Note: There can be several consecutive rows that look identical.

Desired outputs for the examples

Question: In example 3 how do we know whether y0 is a new value or it is y1+dy1

Answer: Other columns (not shown above) in the data set help in identifying the difference.

Note: 0 does not appear in diff, but this is not a difference maker. Hence it is ignored in the solutions.

Properties

1. Dataset contains around 1.3 million rows per day.
2. N is fixed for each row. More than 1 column can be 0.
3. It is known that for a data set exactly one of the following will be true:
   • y0_id > y1_id > …. > yN_id > 0
   • 0 < y0_id < y1_id < …. < yN_id

Solutions (processing 1 day)

Note: Codes are illustrative. They are incomplete!

Brute force

I believe this one doesn’t require explanation. Sample code:

Improvement using pandas

This code is similar to the brute force code.

Improved ‘search’ using dictionary

Idea: look for a matching id and use it for finding difference. Lookup should be quick, so a dictionary can be used with the id value as the key.

Slightly improved ‘search’: using only 1 loop (both id’s are sorted)

Idea: Since both id’s are sorted (assumed as descending order in the code below), a single loop can be used to traverse through both id’s.

Let us iterate through 2 examples:

Example 1:

Iteration 1 (ignore the indexing as Python starts with 0, but column indices start with 1): i=j=0 and the while loop exits at j=0. Id’s match and diff is dy1 (non-zero), so 1 row is appended.

Iteration 2: i=1 and while loop exits at j=1. Id’s match, but diff=0 — therefore no updates. We’re done!

Example 2:

Iteration 1: i=j=0 and the while loop exits at j=0. Id’s don’t match, therefore 1 new row is appended.

Iteration 2: i=1 and the while loop exits at j=0. Id’s match, but diff=0 — therefore no updates. We’re done (kind of, because y2 is not very important in finance \m/)!

Ultimatum: multi-processing

Run-time Comparison

Concatenation — An Additional Problem

Problem

The result is a pandas series of lists. But my desired output is a structure with timestamp + edit in each row. So I used pd.concat(result), which had an additional run time of 3.5 mins (~ clock time for consecutive diff which is a relatively complicated process). This drove me nuts!

Solution

I found that np.concatenate is much faster (~ 40 sec) because of homogeneity. For simplicity I converted timestamp and all col_idx_values to string. The output was written to a CSV file using np.savetxt(“file.csv”, concat_result, fmt=”%s”) or using array_name.tofile(“file.csv”, sep=”,”, format=”%s”).

Learnings

The final solution is a combination of several steps with incremental gains compared to predecessors. Other options such as dask (multiprocessing) were also explored. Ideas related to multi-threading and GPU (CUDA) ended unsuccessfully.

Summary

• Algorithm matters! Gains are in algorithm level. This applies to almost every area of programming.
• Data can be cleaner than it looks. Patterns in the data can help in faster processing. Eg: multiple delimiters, sorted rows, etc.
• Multi-threading in Python is usually not in our hands.
• Multi-processing is cool, but leave 1 core out for safety. Also ensure that there is enough memory to hold a copy of the data frame (+ factor of safety).
• Homogeneous data type can leads to faster processing. However, this is not always true.
• There may be better (faster / memory efficient) solutions.