Menu Close

Autostacker: A Compositional Evolutionary Learning System

Abstract

We introduce an automatic machine learning (AutoML) modeling architecture called Autostacker, which combines an innovative hierarchical stacking architecture and an Evolutionary Algorithm (EA) to perform efficient parameter search. Neither prior domain knowledge about the data nor feature preprocessing is needed. Using EA, Autostacker quickly evolves candidate pipelines with high predictive accuracy. These pipelines can be used as is or as a starting point for human experts to build on. Autostacker finds innovative combinations and structures of machine learning models, rather than selecting a single model and optimizing its hyperparameters. Compared with other AutoML systems on fifteen datasets, Autostacker achieves state-of-art or competitive performance both in terms of test accuracy and time cost.

https://arxiv.org/pdf/1803.00684.pdf

Boyuan Chen Columbia University [email protected] Harvey Wu Columbia University [email protected] Warren Mo University of Chicago [email protected] Ishanu Chattopadhyay University of Chicago [email protected] Hod Lipson Columbia University [email protected]

This paper is really including two previous ideas  in mind, and telling me how to continue my AutoML work:

  1. How to find the best combination of various models
  2. How to take the advantage of the output of various models during training or even testing

The lesson I learned is that never release out your idea till you’re ready!

Lastly,  I would like to keep the source  (Note: In Chinese) the link containing many related AutoML works.

How to be a Winner

Advice for students starting into research work

View this page in Romanian courtesy of Jan Rolsky, Polish courtesy of Nick Stasov, Czech courtesy of Valeria Aleksandrova, Kazakh courtesy of John Vorohovsky, Swedish courtesy of Daniela Milton, Indonesian courtesy of Jordan Silaen.

[N.B.: Observations and recommendations based on first being a UROP student at MIT and later supervising numerous UROP students at MIT and undergraduates and graduate students at UCB.]

Don’t get hung up trying to understand everything at the outset

The biggest challenge you face at the onset of any new project is that there is a huge (seemingly overwhelming) amount of stuff you need to know to tackle your problem properly. While this phenomenon is true in the small for the beginning researcher, it is also true in the large for any research project. So learning how to cope with this challenge is an important skill to master to become a good researcher. In contrast, blocking your action and progress while waiting for complete knowledge is the road to failure.

Coping mechanisms employed by winners include:

  • prioritizing (what do I need to know most)
  • read (everything made available to you, and seek out more; but don’t put months of reading between you and getting started doing things.)
  • multithreading (when blocked on one item or path, is there another I can productively pursue?)
  • pursuing multiple, possible solution techniques (maybe some have easier/less blocks paths than others)
  • wishful thinking (ok, let’s assume this subproblem is solved, does that allow me to go on and solve other problems?)
  • pester people who might have some of the information you need (you might think they should know what you need to know, but often they don’t have a clear idea of what you do and don’t know; start by getting them to give you pointers to things you can use to help yourself. Show respect for their time and always follow up on the resources you’ve been given before asking for a personal explanation.)
  • propose working models — maybe they are wrong or different from others, but they give you something to work with and something concrete to discuss and compare with others. You will refine your models continually, but it’s good to have something concrete in mind to work with.

Losers will stop the first time they run into something they don’t know, cannot solve a problem, or encounter trouble slightly out of what they consider “their part” of the problem and then offer excuses for why they cannot make any progress.
Winners consider the whole problem theirs and look for paths around every hangup.
Losers make sure there is someone or something to blame for their lack of progress.
Winners find ways to make progress despite complications.
Losers know all the reasons it cannot be done
Winners find a way to do it.

Communicate and Synchronize Often

Of course, when you do have to build your own models, solve unexpected problems, make assumptions, etc. do make sure to communicate and synchronize with your fellow researchers. Do they have different models from yours? What can you learn from each others’ models and assumptions? Let them know what you’re thinking, where you’re stuck, and how you’re trying to get around your problems.

Decompose

The whole problem often seems overwhelming. Decompose it into manageable pieces (preferably, with each piece a stable intermediate). Tackle the pieces one at a time. Divide and conquer.

This may sound obvious, but it works. I’ve turned numerous problems which appeared “frightening” in scope into many 1-day or 2-day tasks, and then tackled each nice, contained 1–2 day task as I came to it. As I understood more, new problems and tasks arose, but they could all be broken back down to bite sized pieces which would be tackled one at a time.

Be Organized

In computer systems especially, the biggest limitation to our ability to conquer problems is complexity. You need to work continually to structure the problem and your understanding of it to tackle the inherent complexity. Keep careful track of what you have done and what you need to do. Make lists; write it down; don’t rely on your memory (or worse, yet, your supervisor’s memory) to hold all the things you need to do and all the intermediate issues you need to address.

Prioritize

Make priorities in your efforts and check your priorities with your supervisor. A common occurrence is for your supervisor to ask you to do A, forget about it, and then ask you to do B before you could possibly have finished A. If you are uncertain on whether B should take priority over A, definitely ask. Sometimes it will, but often it won’t, and your supervisor will be glad that you reminded him you were busy solving A. Keep track of B, and when you finish A, see if B still makes sense to pursue.

Realize that your supervisor is busy

Your professor or graduate student supervisor is busy. He hired you to help him get more accomplished than he could have on his own. Your biggest benefit to him is when you can be self moving and motivating.

Do not expect your supervisor to solve all your problems. Find out what he has thought about and suggests for a starting point and work from there. But, realize there may become a time when you have put more quality thought into something than he has (and this will happen more and more often to you as you get into your work). So, when you think you see or know a better way to solve a problem, bring it up. In an ideal scenario this is exactly what should happen. Your supervisor gives you the seed and some directions, then goes off to think about other problems. You put in concentrated time on your problem and ultimately come back with more knowledge and insight into your subproblem than your supervisor.

As a supervisor, I work in two modes:

  1. Until a student has demonstrated that he has thought more deeply about the problem than I have, I strongly advocate that he start things my way.
  2. Once a student has examined a problem in depth, then we can discuss it as peers, and generally the student becomes the expert on this subproblem, and I can offer general advise from my experience and breadth.

Deliver

Once you’ve signed up you have to deliver. But, you do not have to deliver the final solution to everything at once. This, in fact, is a fallacy of many people and research projects.

Losers keep promising a great thing in the future but have nothing to show now.
Winners can show workable/usable results along the way to the solution. These pieces can include:

  • solutions to simplified models
  • pieces of a flow
  • intermediate output/data
  • measurements of problem characteristics
  • stable intermediates (see below)

Demonstrate progress. This allows your supervisor to offer early feedback and to help you prioritize your attention—this will often help you both make mid-course corrections increasing the likelihood you will end up with interesting results in the end. Requirements and understanding invariable evolve (remember the key challenge at the beginning is incompletely knowledge). Change and redirection is normal, expected, and healthy (since it is usually a result of greater knowledge and understanding). The incremental model is robust and prepared for this adaptation while the monolithic (all-at-once) model is brittle and often leads to great solutions which don’t address the real problem.

Incrementally grow your solutions (especially software). In the new chapters which appear in the 20th Anniversary edition of Mythical Man Month, Brooks identifies incremental development and progressive refinement towards the goal as one of the best, new techniques which he’s come to appreciate since the original writing of MMM. From my own experience, I whole heartedly agree with this, and it does have a very positive impact on morale (yours, your team’s, your supervisor’s).

Target stable intermediates

Look for stable intermediate points on your incremental path to solving some problem.

  • points where some clear piece of the problem has been solved (has a nice interface to this subproblem, produces results at this stage)
  • things you can build upon
  • things you can spin-out
  • things you can share with team members (allow them to help)
  • points of accomplishment

Don’t turn problems (subtasks) into research problems unnecessarily.

Often you’ll run into a subtask with no single, obviously right solution. If solving this piece right is key to the overall goals, maybe it will be necessary to devote time to studying and solving this subproblem better than it has ever been solved before. However, for most sub-problems, this is not the case. You want to keep focused on the overall goals of the project and come up with an “adequate” solution for this problem. In general, try to do the obvious or simple thing which can be done expediently. Make notes on the possible weaknesses and the alternatives you could explore should these weakness prove limiting. Then, if this does become a bottleneck or weak link in the solution chain, you can revisit it and your alternatives and invest more effort exploring them.

Learn to solve your own problems

In general, in life, there won’t always be someone to turn to who has all the answers. It is vitally important that you learn how to tackle all the kinds of problems you may encounter. Use your supervisors as a crutch or scaffolding only to get yourself started. Watch them and learn not just the answers they help you find, but how they find the answers you were unable to obtain on your own. Strive for independence. Learn techniques and gain confidence in your own ability to solve problems now. [source]


Non-Stationary Texture Synthesis by Adversarial Expansion

Abstract

The real world exhibits an abundance of non-stationary textures. Examples include textures with large-scale structures, as well as spatially variant and inhomogeneous textures. While existing example-based texture synthesis methods can cope well with stationary textures, non-stationary textures still pose a considerable challenge, which remains unresolved. In this paper, we propose a new approach for example-based non-stationary texture synthesis. Our approach uses a generative adversarial network (GAN), trained to double the spatial extent of texture blocks extracted from a specific texture exemplary. Once trained, the fully convolutional generator is able to expand the size of the entire exemplar, as well as of any of its sub-blocks. We demonstrate that this conceptually simple approach is highly effective for capturing large-scale structures, as well as other non-stationary attributes of the input exemplary. As a result, it can cope with challenging textures, which, to our knowledge, no other existing method can handle.

CCS Concepts: • Computing methodologies → Appearance and texture representations; Image manipulation; Texturing;

YANG ZHOU, Shenzhen University and Huazhong University of Science & Technology ZHEN ZHU and XIANG BAI, Huazhong University of Science and Technology DANI LISCHINSKI, The Hebrew University of Jerusalem DANIEL COHEN-OR, Shenzhen University and Tel Aviv University HUI HUANG, Shenzhen University

https://arxiv.org/pdf/1805.04487.pdf

This work is the most amazing one that I have never seen.   I have some minds about it:

  1. How about to use it do microscopy image synthesis?
  2. It is definitely able to create customized production decorated with paint arts.
  3. Something else?  see this video