• 38 Posts
  • 58 Comments
Joined 6 months ago
cake
Cake day: April 4th, 2024

help-circle



















  • merari42@lemmy.worldtoScience Memes@mander.xyzPandas
    link
    fedilink
    English
    arrow-up
    4
    ·
    2 months ago

    Have you heard that there are great serialised file formats like .parquet from appache arrow, that can easily be used in typical data science packages like duckdb or polars. Perhaps it even works with pandas (although do not know it that well. I avoid pandas as much as possible as someone who comes from the R tidyverse and try to use polars more when I work in python, because it often feels more intuitive to work with for me.)



  • I have a ten-year old MacBook Pro with an i7 and 16gb of ram. Just because this thing was a total beast when it was new does not mean it isn’t old now. works great with Ubuntu though. It’s still not a good idea to run it as a server though. My raspberry pi consumes a lot less energy for some basic web hosting tasks. I only use the old MBP to run memory intense docker containers like openrouteservice and I guess just using some hosting service for that would not be much more expensive.


  • Depends on what you do with it. Synthetic data seems to be really powerful if it’s human controlled and well built. Stuff like tiny stories (simple llm-generated stories that only use the complexity of a 3-year olds vocabulary) can be used to make tiny language models produce sensible English output. My favourite newer example is the base data for AlphaProof (llm-generated translations of proofs in Math-Papers to the proof-validation system LEAN) to teach an LLM the basic structure of Mathematics proofs. The validation in LEAN itself can be used to only keep high-quality (i.e. correct) proofs. Since AlphaProof is basically a reinforcement learning routine that uses an llm to generate good ideas for proof steps to reduce the size of the space of proof steps, applying it yields new correct proofs that can be used to further improve its internal training data.